{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "13_Recurrent_Neural_Networks",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "bOChJSNXtC9g",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Recurrent Neural Networks"
      ]
    },
    {
      "metadata": {
        "id": "OLIxEDq6VhvZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/logo.png\" width=150>\n",
        "\n",
        "When working with sequential data (time-series, sentences, etc.) the order of the inputs is crucial for the task at hand. Recurrent neural networks (RNNs) process sequential data by accounting for the current input and also what has been learned from previous inputs. In this notebook, we'll learn how to create and train RNNs on sequential data.\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/rnn.png\" width=550>\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "VoMq0eFRvugb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Overview"
      ]
    },
    {
      "metadata": {
        "id": "qWro5T5qTJJL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* **Objective:**  Process sequential data by accounting for the currend input and also what has been learned from previous inputs.\n",
        "* **Advantages:** \n",
        "    * Account for order and previous inputs in a meaningful way.\n",
        "    * Conditioned generation for generating sequences.\n",
        "* **Disadvantages:** \n",
        "    * Each time step's prediction depends on the previous prediction so it's difficult to parallelize RNN operations. \n",
        "    * Processing long sequences can yield memory and computation issues.\n",
        "    * Interpretability is difficult but there are few [techniques](https://arxiv.org/abs/1506.02078) that use the activations from RNNs to see what parts of the inputs are processed. \n",
        "* **Miscellaneous:** \n",
        "    * Architectural tweaks to make RNNs faster and interpretable is an ongoing area of research."
      ]
    },
    {
      "metadata": {
        "id": "rsHeBbehrKzl",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/rnn2.png\" width=650>\n",
        "\n",
        "RNN forward pass for a single time step $X_t$:\n",
        "\n",
        "$h_t = tanh(W_{hh}h_{t-1} + W_{xh}X_t+b_h)$\n",
        "\n",
        "$y_t = W_{hy}h_t + b_y $\n",
        "\n",
        "$ P(y) = softmax(y_t) = \\frac{e^y}{\\sum e^y} $\n",
        "\n",
        "*where*:\n",
        "* $X_t$ = input at time step t | $\\in \\mathbb{R}^{NXE}$ ($N$ is the batch size, $E$ is the embedding dim)\n",
        "* $W_{hh}$ = hidden units weights| $\\in \\mathbb{R}^{HXH}$ ($H$ is the hidden dim)\n",
        "* $h_{t-1}$ = previous timestep's hidden state $\\in \\mathbb{R}^{NXH}$\n",
        "* $W_{xh}$ = input weights| $\\in \\mathbb{R}^{EXH}$\n",
        "* $b_h$ = hidden units bias $\\in \\mathbb{R}^{HX1}$\n",
        "* $W_{hy}$ = output weights| $\\in \\mathbb{R}^{HXC}$ ($C$ is the number of classes)\n",
        "* $b_y$ = output bias $\\in \\mathbb{R}^{CX1}$\n",
        "\n",
        "You repeat this for every time step's input ($X_{t+1}, X_{t+2}, ..., X_{N})$ to the get the predicted outputs at each time step.\n",
        "\n",
        "**Note**: At the first time step, the previous hidden state $h_{t-1}$ can either be a zero vector (unconditioned) or initialize (conditioned). If we are conditioning the RNN, the first hidden state $h_0$ can belong to a specific condition or we can concat the specific condition to the randomly initialized hidden vectors at each time step. More on this in the subsequent notebooks on RNNs."
      ]
    },
    {
      "metadata": {
        "id": "dIXlGMExJD6w",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Let's see what the forward pass looks like with an RNN for a synthetic task such as processing reviews (a sequence of words) to predict the sentiment at the end of processing the review."
      ]
    },
    {
      "metadata": {
        "id": "RcWE5cw0_cKA",
        "colab_type": "code",
        "outputId": "2250dd7c-0d07-4d97-ff14-2448cd5b09d0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 139
        }
      },
      "cell_type": "code",
      "source": [
        "# Load PyTorch library\n",
        "!pip3 install torch"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting torch\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7e/60/66415660aa46b23b5e1b72bc762e816736ce8d7260213e22365af51e8f9c/torch-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (591.8MB)\n",
            "\u001b[K    100% |████████████████████████████████| 591.8MB 26kB/s \n",
            "tcmalloc: large alloc 1073750016 bytes == 0x623ec000 @  0x7fe65475b2a4 0x591a07 0x5b5d56 0x502e9a 0x506859 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x504c28 0x502540 0x502f3d 0x507641\n",
            "\u001b[?25hInstalling collected packages: torch\n",
            "Successfully installed torch-1.0.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "o6eEK1wM_dXG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Qi9hIEV6COLF",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "batch_size = 5\n",
        "seq_size = 10 # max length per input (masking will be used for sequences that aren't this max length)\n",
        "x_lengths = [8, 5, 4, 10, 5] # lengths of each input sequence\n",
        "embedding_dim = 100\n",
        "rnn_hidden_dim = 256\n",
        "output_dim = 4"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "bLEzfxjhB94C",
        "colab_type": "code",
        "outputId": "a2952225-5c5f-4250-ec79-6288dbf34c42",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Initialize synthetic inputs\n",
        "x_in = torch.randn(batch_size, seq_size, embedding_dim)\n",
        "x_lengths = torch.tensor(x_lengths)\n",
        "print (x_in.size())"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([5, 10, 100])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "dr6oLqtXB98N",
        "colab_type": "code",
        "outputId": "b54ff9e2-2857-499c-b739-c2c15e869977",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Initialize hidden state\n",
        "hidden_t = torch.zeros((batch_size, rnn_hidden_dim))\n",
        "print (hidden_t.size())"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([5, 256])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ryZMOLLgB9-v",
        "colab_type": "code",
        "outputId": "1bf649dd-4c47-4686-f15e-a9b408e32e61",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Initialize RNN cell\n",
        "rnn_cell = nn.RNNCell(embedding_dim, rnn_hidden_dim)\n",
        "print (rnn_cell)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "RNNCell(100, 256)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "rlbZ7ujxExXb",
        "colab_type": "code",
        "outputId": "9d6be9d0-20bc-4be8-c155-8d098ce5cbe1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Forward pass through RNN\n",
        "x_in = x_in.permute(1, 0, 2) # RNN needs batch_size to be at dim 1\n",
        "\n",
        "# Loop through the inputs time steps\n",
        "hiddens = []\n",
        "for t in range(seq_size):\n",
        "    hidden_t = rnn_cell(x_in[t], hidden_t)\n",
        "    hiddens.append(hidden_t)\n",
        "hiddens = torch.stack(hiddens)\n",
        "hiddens = hiddens.permute(1, 0, 2) # bring batch_size back to dim 0\n",
        "print (hiddens.size())"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([5, 10, 256])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "3TTL-jmg-MHa",
        "colab_type": "code",
        "outputId": "22524f37-2dd4-472e-b76d-404eed0250a7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# We also could've used a more abstracted layer\n",
        "x_in = torch.randn(batch_size, seq_size, embedding_dim)\n",
        "rnn = nn.RNN(embedding_dim, rnn_hidden_dim, batch_first=True)\n",
        "out, h_n = rnn(x_in) #h_n is the last hidden state\n",
        "print (\"out: \", out.size())\n",
        "print (\"h_n: \", h_n.size())"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "out:  torch.Size([5, 10, 256])\n",
            "h_n:  torch.Size([1, 5, 256])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "iAsyRNnbHwcT",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def gather_last_relevant_hidden(hiddens, x_lengths):\n",
        "    x_lengths = x_lengths.long().detach().cpu().numpy() - 1\n",
        "    out = []\n",
        "    for batch_index, column_index in enumerate(x_lengths):\n",
        "        out.append(hiddens[batch_index, column_index])\n",
        "    return torch.stack(out)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PVhp1KLqHqpA",
        "colab_type": "code",
        "outputId": "f4b0708b-88dc-4b28-93c0-073bbcb6fe2a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Gather the last relevant hidden state\n",
        "z = gather_last_relevant_hidden(hiddens, x_lengths)\n",
        "print (z.size())"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([5, 256])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "yGk_iZ5cITZl",
        "colab_type": "code",
        "outputId": "082e0035-4b31-4066-f02a-6686e2070bdd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "cell_type": "code",
      "source": [
        "# Forward pass through FC layer\n",
        "fc1 = nn.Linear(rnn_hidden_dim, output_dim)\n",
        "y_pred = fc1(z)\n",
        "y_pred = F.softmax(y_pred, dim=1)\n",
        "print (y_pred.size())\n",
        "print (y_pred)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([5, 4])\n",
            "tensor([[0.2963, 0.2147, 0.2409, 0.2482],\n",
            "        [0.2638, 0.3373, 0.1835, 0.2154],\n",
            "        [0.2820, 0.2741, 0.1941, 0.2498],\n",
            "        [0.1895, 0.3276, 0.2353, 0.2476],\n",
            "        [0.2600, 0.2098, 0.2310, 0.2992]], grad_fn=<SoftmaxBackward>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "hPBQpki_n6yY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Sequential data"
      ]
    },
    {
      "metadata": {
        "id": "kP1awuluoCSr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "There are a variety of different sequential tasks that RNNs can help with.\n",
        "\n",
        "1. **One to one**: there is one input and produces one output. \n",
        "    * Ex. Given a word predict it's class (verb, noun, etc.).\n",
        "2. **One to many**: one input generates many outputs.\n",
        "    * Ex. Given a sentiment (positive, negative, etc.) generate a review.\n",
        "3. **Many to one**: Many inputs are sequentially processed to generate one output.\n",
        "    * Ex. Process the words in a review to predict the sentiment.\n",
        "4. **Many to many**: Many inputs are sequentially processed to generate many outputs.\n",
        "    * Ex. Given a sentence in French, processes the entire sentence and then generate the English translation.\n",
        "    * Ex. Given a sequence of time-series data, predict the probability of an event (risk of disease) at each time step.\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/seq2seq.jpeg\" width=700>"
      ]
    },
    {
      "metadata": {
        "id": "tnxUIEMdukYY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Issues with vanilla RNNs"
      ]
    },
    {
      "metadata": {
        "id": "uMx2s93VLUTt",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "There are several issues with the vanilla RNN that we've seen so far. \n",
        "\n",
        "1. When we have an input sequence that has many time steps, it becomes difficult for the model to retain information seen earlier as we process more and more of the downstream timesteps. The goals of the model is to retain the useful components in the previously seen time steps but this becomes cumbersome when we have so many time steps to process. \n",
        "\n",
        "2. During backpropagation, the gradient from the loss has to travel all the way back towards the first time step. If our gradient is larger than 1 (${1.01}^{1000} = 20959$) or less than 1 (${0.99}^{1000} = 4.31e-5$) and we have lot's of time steps, this can quickly spiral out of control.\n",
        "\n",
        "To address both these issues, the concept of gating was introduced to RNNs. Gating allows RNNs to control the information flow between each time step to optimize on the task. Selectively allowing information to pass through allows the model to process inputs with many time steps. The most common RNN gated varients are the long short term memory ([LSTM](https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM)) units and gated recurrent units ([GRUs](https://pytorch.org/docs/stable/nn.html#torch.nn.GRU)). You can read more about how these units work [here](http://colah.github.io/posts/2015-08-Understanding-LSTMs/).\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/gates.png\" width=900>"
      ]
    },
    {
      "metadata": {
        "id": "tirko0kwp-9J",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# GRU in PyTorch\n",
        "gru = nn.GRU(input_size=embedding_dim, hidden_size=rnn_hidden_dim, \n",
        "             batch_first=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "UZjUhh4VBWxM",
        "colab_type": "code",
        "outputId": "07a0ab92-2e80-458d-f81d-782450b9094b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Initialize synthetic input\n",
        "x_in = torch.randn(batch_size, seq_size, embedding_dim)\n",
        "print (x_in.size())"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([5, 10, 100])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "xJ_SE7AvBfa4",
        "colab_type": "code",
        "outputId": "fb4cb2b3-72a2-4609-985c-0b23a1decdbb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# Forward pass\n",
        "out, h_n = gru(x_in)\n",
        "print (\"out:\", out.size())\n",
        "print (\"h_n:\", h_n.size())"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "out: torch.Size([5, 10, 256])\n",
            "h_n: torch.Size([1, 5, 256])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ij_GA2Rr9BbA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Note**: Choosing whether to use GRU or LSTM really depends on the data and empirical performance. GRUs offer comparable performance with reduce number of parameters while LSTMs are more efficient and may make the difference in performance for your particular task."
      ]
    },
    {
      "metadata": {
        "id": "9agJw4gwK1LC",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Bidirectional RNNs"
      ]
    },
    {
      "metadata": {
        "id": "Xck0n-KpmXkV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "There have been many advancements with RNNs ([attention](https://www.oreilly.com/ideas/interpretability-via-attentional-and-memory-based-interfaces-using-tensorflow), Quasi RNNs, etc.) that we will cover in later lessons but one of the basic and widely used ones are bidirectional RNNs (Bi-RNNs). The motivation behind bidirectional RNNs is to process an input sequence by both directions. Accounting for context from both sides can aid in performance when the entire input sequence is known at time of inference. A common application of Bi-RNNs is in translation where it's advantageous to look at an entire sentence from both sides when translating to another language (ie. Japanese → English).\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/birnn.png\" width=700>"
      ]
    },
    {
      "metadata": {
        "id": "gSk_5XrvApCd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# BiGRU in PyTorch\n",
        "bi_gru = nn.GRU(input_size=embedding_dim, hidden_size=rnn_hidden_dim, \n",
        "                batch_first=True, bidirectional=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Fx7-GTptBCtZ",
        "colab_type": "code",
        "outputId": "6150454c-e67c-4b9f-e60e-6b963916ce05",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# Forward pass\n",
        "out, h_n = bi_gru(x_in)\n",
        "print (\"out:\", out.size()) # collection of all hidden states from the RNN for each time step\n",
        "print (\"h_n:\", h_n.size()) # last hidden state from the RNN"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "out: torch.Size([5, 10, 512])\n",
            "h_n: torch.Size([2, 5, 256])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "k5lvJirLBjI6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Notice that the output for each sample at each timestamp has size 512 (double the hidden dim). This is because this includes both the forward and backward directions from the BiRNN. "
      ]
    },
    {
      "metadata": {
        "id": "mJSknbofK2S9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Document classification with RNNs"
      ]
    },
    {
      "metadata": {
        "id": "JgYdEZmHlmft",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Let's apply RNNs to the document classification task from the [emebddings notebook](https://colab.research.google.com/drive/1yDa5ZTqKVoLl-qRgH-N9xs3pdrDJ0Fb4) where we want to predict an article's category given its title."
      ]
    },
    {
      "metadata": {
        "id": "eIvXqvPQEiDC",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Set up"
      ]
    },
    {
      "metadata": {
        "id": "muTcvMynlmAu",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import os\n",
        "from argparse import Namespace\n",
        "import collections\n",
        "import copy\n",
        "import json\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import re\n",
        "import torch"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "00ESjecep-_y",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Set Numpy and PyTorch seeds\n",
        "def set_seeds(seed, cuda):\n",
        "    np.random.seed(seed)\n",
        "    torch.manual_seed(seed)\n",
        "    if cuda:\n",
        "        torch.cuda.manual_seed_all(seed)\n",
        "        \n",
        "# Creating directories\n",
        "def create_dirs(dirpath):\n",
        "    if not os.path.exists(dirpath):\n",
        "        os.makedirs(dirpath)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "m67THDvxEl1e",
        "colab_type": "code",
        "outputId": "414a482e-817b-4940-92ec-229596343b7e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Arguments\n",
        "args = Namespace(\n",
        "    seed=1234,\n",
        "    cuda=True,\n",
        "    shuffle=True,\n",
        "    data_file=\"news.csv\",\n",
        "    vectorizer_file=\"vectorizer.json\",\n",
        "    model_state_file=\"model.pth\",\n",
        "    save_dir=\"news\",\n",
        "    train_size=0.7,\n",
        "    val_size=0.15,\n",
        "    test_size=0.15,\n",
        "    pretrained_embeddings=None,\n",
        "    cutoff=25, # token must appear at least <cutoff> times to be in SequenceVocabulary\n",
        "    num_epochs=5,\n",
        "    early_stopping_criteria=5,\n",
        "    learning_rate=1e-3,\n",
        "    batch_size=64,\n",
        "    embedding_dim=100,\n",
        "    rnn_hidden_dim=128,\n",
        "    hidden_dim=100,\n",
        "    num_layers=1,\n",
        "    bidirectional=False,\n",
        "    dropout_p=0.1,\n",
        ")\n",
        "\n",
        "# Set seeds\n",
        "set_seeds(seed=args.seed, cuda=args.cuda)\n",
        "\n",
        "# Create save dir\n",
        "create_dirs(args.save_dir)\n",
        "\n",
        "# Expand filepaths\n",
        "args.vectorizer_file = os.path.join(args.save_dir, args.vectorizer_file)\n",
        "args.model_state_file = os.path.join(args.save_dir, args.model_state_file)\n",
        "\n",
        "# Check CUDA\n",
        "if not torch.cuda.is_available():\n",
        "    args.cuda = False\n",
        "args.device = torch.device(\"cuda\" if args.cuda else \"cpu\")\n",
        "print(\"Using CUDA: {}\".format(args.cuda))"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using CUDA: True\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "s7T-_kGvExVW",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Data"
      ]
    },
    {
      "metadata": {
        "id": "XVyK25xOEwjN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import re\n",
        "import urllib"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "M_gclwECEwll",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Upload data from GitHub to notebook's local drive\n",
        "url = \"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/data/news.csv\"\n",
        "response = urllib.request.urlopen(url)\n",
        "html = response.read()\n",
        "with open(args.data_file, 'wb') as fp:\n",
        "    fp.write(html)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "V244zOIPEwoP",
        "colab_type": "code",
        "outputId": "6c7f9959-362b-43db-80d9-807f9595c4c1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "cell_type": "code",
      "source": [
        "# Raw data\n",
        "df = pd.read_csv(args.data_file, header=0)\n",
        "df.head()"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>category</th>\n",
              "      <th>title</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Business</td>\n",
              "      <td>Wall St. Bears Claw Back Into the Black (Reuters)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Business</td>\n",
              "      <td>Carlyle Looks Toward Commercial Aerospace (Reu...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Business</td>\n",
              "      <td>Oil and Economy Cloud Stocks' Outlook (Reuters)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Business</td>\n",
              "      <td>Iraq Halts Oil Exports from Main Southern Pipe...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Business</td>\n",
              "      <td>Oil prices soar to all-time record, posing new...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   category                                              title\n",
              "0  Business  Wall St. Bears Claw Back Into the Black (Reuters)\n",
              "1  Business  Carlyle Looks Toward Commercial Aerospace (Reu...\n",
              "2  Business    Oil and Economy Cloud Stocks' Outlook (Reuters)\n",
              "3  Business  Iraq Halts Oil Exports from Main Southern Pipe...\n",
              "4  Business  Oil prices soar to all-time record, posing new..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "metadata": {
        "id": "ICl2MNK4EwrL",
        "colab_type": "code",
        "outputId": "0669edde-c740-457a-e342-70d10e485c72",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# Split by category\n",
        "by_category = collections.defaultdict(list)\n",
        "for _, row in df.iterrows():\n",
        "    by_category[row.category].append(row.to_dict())\n",
        "for category in by_category:\n",
        "    print (\"{0}: {1}\".format(category, len(by_category[category])))"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Business: 30000\n",
            "Sci/Tech: 30000\n",
            "Sports: 30000\n",
            "World: 30000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "76PwKQHLEww5",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Create split data\n",
        "final_list = []\n",
        "for _, item_list in sorted(by_category.items()):\n",
        "    if args.shuffle:\n",
        "        np.random.shuffle(item_list)\n",
        "    n = len(item_list)\n",
        "    n_train = int(args.train_size*n)\n",
        "    n_val = int(args.val_size*n)\n",
        "    n_test = int(args.test_size*n)\n",
        "\n",
        "  # Give data point a split attribute\n",
        "    for item in item_list[:n_train]:\n",
        "        item['split'] = 'train'\n",
        "    for item in item_list[n_train:n_train+n_val]:\n",
        "        item['split'] = 'val'\n",
        "    for item in item_list[n_train+n_val:]:\n",
        "        item['split'] = 'test'  \n",
        "\n",
        "    # Add to final list\n",
        "    final_list.extend(item_list)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "CQeS0KHOEwzm",
        "colab_type": "code",
        "outputId": "a2a8ed57-da3c-4441-9159-08e3d66e1a80",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# df with split datasets\n",
        "split_df = pd.DataFrame(final_list)\n",
        "split_df[\"split\"].value_counts()"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "train    84000\n",
              "val      18000\n",
              "test     18000\n",
              "Name: split, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "metadata": {
        "id": "pPJDyVusEw3-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "63051d89-3ea7-44d1-cc5a-5655e904507f"
      },
      "cell_type": "code",
      "source": [
        "# Preprocessing\n",
        "def preprocess_text(text):\n",
        "    text = ' '.join(word.lower() for word in text.split(\" \"))\n",
        "    text = re.sub(r\"([.,!?])\", r\" \\1 \", text)\n",
        "    text = re.sub(r\"[^a-zA-Z.,!?]+\", r\" \", text)\n",
        "    text = text.strip()\n",
        "    return text\n",
        "    \n",
        "split_df.title = split_df.title.apply(preprocess_text)\n",
        "split_df.head()"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>category</th>\n",
              "      <th>split</th>\n",
              "      <th>title</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Business</td>\n",
              "      <td>train</td>\n",
              "      <td>general electric posts higher rd quarter profit</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Business</td>\n",
              "      <td>train</td>\n",
              "      <td>lilly to eliminate up to us jobs</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Business</td>\n",
              "      <td>train</td>\n",
              "      <td>s amp p lowers america west outlook to negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Business</td>\n",
              "      <td>train</td>\n",
              "      <td>does rand walk the talk on labor policy ?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Business</td>\n",
              "      <td>train</td>\n",
              "      <td>housekeeper advocates for changes</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   category  split                                            title\n",
              "0  Business  train  general electric posts higher rd quarter profit\n",
              "1  Business  train                 lilly to eliminate up to us jobs\n",
              "2  Business  train  s amp p lowers america west outlook to negative\n",
              "3  Business  train        does rand walk the talk on labor policy ?\n",
              "4  Business  train                housekeeper advocates for changes"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "metadata": {
        "id": "NHzGXAI3E7lF",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Vocabulary"
      ]
    },
    {
      "metadata": {
        "id": "ZIRUjX0MEw88",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class Vocabulary(object):\n",
        "    def __init__(self, token_to_idx=None):\n",
        "\n",
        "        # Token to index\n",
        "        if token_to_idx is None:\n",
        "            token_to_idx = {}\n",
        "        self.token_to_idx = token_to_idx\n",
        "\n",
        "        # Index to token\n",
        "        self.idx_to_token = {idx: token \\\n",
        "                             for token, idx in self.token_to_idx.items()}\n",
        "\n",
        "    def to_serializable(self):\n",
        "        return {'token_to_idx': self.token_to_idx}\n",
        "\n",
        "    @classmethod\n",
        "    def from_serializable(cls, contents):\n",
        "        return cls(**contents)\n",
        "\n",
        "    def add_token(self, token):\n",
        "        if token in self.token_to_idx:\n",
        "            index = self.token_to_idx[token]\n",
        "        else:\n",
        "            index = len(self.token_to_idx)\n",
        "            self.token_to_idx[token] = index\n",
        "            self.idx_to_token[index] = token\n",
        "        return index\n",
        "\n",
        "    def add_tokens(self, tokens):\n",
        "        return [self.add_token[token] for token in tokens]\n",
        "\n",
        "    def lookup_token(self, token):\n",
        "        return self.token_to_idx[token]\n",
        "\n",
        "    def lookup_index(self, index):\n",
        "        if index not in self.idx_to_token:\n",
        "            raise KeyError(\"the index (%d) is not in the Vocabulary\" % index)\n",
        "        return self.idx_to_token[index]\n",
        "\n",
        "    def __str__(self):\n",
        "        return \"<Vocabulary(size=%d)>\" % len(self)\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.token_to_idx)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "1LtYf3lpExBb",
        "colab_type": "code",
        "outputId": "1d2f3947-6662-4dec-8df7-724967589d3a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# Vocabulary instance\n",
        "category_vocab = Vocabulary()\n",
        "for index, row in df.iterrows():\n",
        "    category_vocab.add_token(row.category)\n",
        "print (category_vocab) # __str__\n",
        "print (len(category_vocab)) # __len__\n",
        "index = category_vocab.lookup_token(\"Business\")\n",
        "print (index)\n",
        "print (category_vocab.lookup_index(index))"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<Vocabulary(size=4)>\n",
            "4\n",
            "0\n",
            "Business\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Z0zkF6CsE_yH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Sequence vocabulary"
      ]
    },
    {
      "metadata": {
        "id": "QtntaISyE_1c",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Next, we're going to create our Vocabulary classes for the article's title, which is a sequence of tokens."
      ]
    },
    {
      "metadata": {
        "id": "ovI8QRefEw_p",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from collections import Counter\n",
        "import string"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4W3ZouuTEw1_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class SequenceVocabulary(Vocabulary):\n",
        "    def __init__(self, token_to_idx=None, unk_token=\"<UNK>\",\n",
        "                 mask_token=\"<MASK>\", begin_seq_token=\"<BEGIN>\",\n",
        "                 end_seq_token=\"<END>\"):\n",
        "\n",
        "        super(SequenceVocabulary, self).__init__(token_to_idx)\n",
        "\n",
        "        self.mask_token = mask_token\n",
        "        self.unk_token = unk_token\n",
        "        self.begin_seq_token = begin_seq_token\n",
        "        self.end_seq_token = end_seq_token\n",
        "\n",
        "        self.mask_index = self.add_token(self.mask_token)\n",
        "        self.unk_index = self.add_token(self.unk_token)\n",
        "        self.begin_seq_index = self.add_token(self.begin_seq_token)\n",
        "        self.end_seq_index = self.add_token(self.end_seq_token)\n",
        "        \n",
        "        # Index to token\n",
        "        self.idx_to_token = {idx: token \\\n",
        "                             for token, idx in self.token_to_idx.items()}\n",
        "\n",
        "    def to_serializable(self):\n",
        "        contents = super(SequenceVocabulary, self).to_serializable()\n",
        "        contents.update({'unk_token': self.unk_token,\n",
        "                         'mask_token': self.mask_token,\n",
        "                         'begin_seq_token': self.begin_seq_token,\n",
        "                         'end_seq_token': self.end_seq_token})\n",
        "        return contents\n",
        "\n",
        "    def lookup_token(self, token):\n",
        "        return self.token_to_idx.get(token, self.unk_index)\n",
        "    \n",
        "    def lookup_index(self, index):\n",
        "        if index not in self.idx_to_token:\n",
        "            raise KeyError(\"the index (%d) is not in the SequenceVocabulary\" % index)\n",
        "        return self.idx_to_token[index]\n",
        "    \n",
        "    def __str__(self):\n",
        "        return \"<SequenceVocabulary(size=%d)>\" % len(self.token_to_idx)\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.token_to_idx)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "g5UHjpi3El37",
        "colab_type": "code",
        "outputId": "4ca7dd37-112e-4839-becf-6a42ed3716fb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# Get word counts\n",
        "word_counts = Counter()\n",
        "for title in split_df.title:\n",
        "    for token in title.split(\" \"):\n",
        "        if token not in string.punctuation:\n",
        "            word_counts[token] += 1\n",
        "\n",
        "# Create SequenceVocabulary instance\n",
        "title_vocab = SequenceVocabulary()\n",
        "for word, word_count in word_counts.items():\n",
        "    if word_count >= args.cutoff:\n",
        "        title_vocab.add_token(word)\n",
        "print (title_vocab) # __str__\n",
        "print (len(title_vocab)) # __len__\n",
        "index = title_vocab.lookup_token(\"general\")\n",
        "print (index)\n",
        "print (title_vocab.lookup_index(index))"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<SequenceVocabulary(size=4400)>\n",
            "4400\n",
            "4\n",
            "general\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "4Dag6H0SFHAG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Vectorizer"
      ]
    },
    {
      "metadata": {
        "id": "VQIfxcUuKwzz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Something new that we introduce in this Vectorizer is calculating the length of our input sequence. We will use this later on to extract the last relevant hidden state for each input sequence."
      ]
    },
    {
      "metadata": {
        "id": "tsNtEnhBEl6s",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class NewsVectorizer(object):\n",
        "    def __init__(self, title_vocab, category_vocab):\n",
        "        self.title_vocab = title_vocab\n",
        "        self.category_vocab = category_vocab\n",
        "\n",
        "    def vectorize(self, title):\n",
        "        indices = [self.title_vocab.lookup_token(token) for token in title.split(\" \")]\n",
        "        indices = [self.title_vocab.begin_seq_index] + indices + \\\n",
        "            [self.title_vocab.end_seq_index]\n",
        "        \n",
        "        # Create vector\n",
        "        title_length = len(indices)\n",
        "        vector = np.zeros(title_length, dtype=np.int64)\n",
        "        vector[:len(indices)] = indices\n",
        "\n",
        "        return vector, title_length\n",
        "    \n",
        "    def unvectorize(self, vector):\n",
        "        tokens = [self.title_vocab.lookup_index(index) for index in vector]\n",
        "        title = \" \".join(token for token in tokens)\n",
        "        return title\n",
        "\n",
        "    @classmethod\n",
        "    def from_dataframe(cls, df, cutoff):\n",
        "        \n",
        "        # Create class vocab\n",
        "        category_vocab = Vocabulary()        \n",
        "        for category in sorted(set(df.category)):\n",
        "            category_vocab.add_token(category)\n",
        "\n",
        "        # Get word counts\n",
        "        word_counts = Counter()\n",
        "        for title in df.title:\n",
        "            for token in title.split(\" \"):\n",
        "                word_counts[token] += 1\n",
        "        \n",
        "        # Create title vocab\n",
        "        title_vocab = SequenceVocabulary()\n",
        "        for word, word_count in word_counts.items():\n",
        "            if word_count >= cutoff:\n",
        "                title_vocab.add_token(word)\n",
        "        \n",
        "        return cls(title_vocab, category_vocab)\n",
        "\n",
        "    @classmethod\n",
        "    def from_serializable(cls, contents):\n",
        "        title_vocab = SequenceVocabulary.from_serializable(contents['title_vocab'])\n",
        "        category_vocab = Vocabulary.from_serializable(contents['category_vocab'])\n",
        "        return cls(title_vocab=title_vocab, category_vocab=category_vocab)\n",
        "    \n",
        "    def to_serializable(self):\n",
        "        return {'title_vocab': self.title_vocab.to_serializable(),\n",
        "                'category_vocab': self.category_vocab.to_serializable()}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "JtRRXU53El9Y",
        "colab_type": "code",
        "outputId": "47ddcba1-de72-45b1-ef72-9cf21050abb5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "cell_type": "code",
      "source": [
        "# Vectorizer instance\n",
        "vectorizer = NewsVectorizer.from_dataframe(split_df, cutoff=args.cutoff)\n",
        "print (vectorizer.title_vocab)\n",
        "print (vectorizer.category_vocab)\n",
        "vectorized_title, title_length = vectorizer.vectorize(preprocess_text(\n",
        "    \"Roger Federer wins the Wimbledon tennis tournament.\"))\n",
        "print (np.shape(vectorized_title))\n",
        "print (\"title_length:\", title_length)\n",
        "print (vectorized_title)\n",
        "print (vectorizer.unvectorize(vectorized_title))"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<SequenceVocabulary(size=4404)>\n",
            "<Vocabulary(size=4)>\n",
            "(10,)\n",
            "title_length: 10\n",
            "[   2    1 4151 1231   25    1 2392 4076   38    3]\n",
            "<BEGIN> <UNK> federer wins the <UNK> tennis tournament . <END>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "uk_QvpVfFM0S",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Dataset"
      ]
    },
    {
      "metadata": {
        "id": "oU7oDdelFMR9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from torch.utils.data import Dataset, DataLoader"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pB7FHmiSFMXA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class NewsDataset(Dataset):\n",
        "    def __init__(self, df, vectorizer):\n",
        "        self.df = df\n",
        "        self.vectorizer = vectorizer\n",
        "\n",
        "        # Data splits\n",
        "        self.train_df = self.df[self.df.split=='train']\n",
        "        self.train_size = len(self.train_df)\n",
        "        self.val_df = self.df[self.df.split=='val']\n",
        "        self.val_size = len(self.val_df)\n",
        "        self.test_df = self.df[self.df.split=='test']\n",
        "        self.test_size = len(self.test_df)\n",
        "        self.lookup_dict = {'train': (self.train_df, self.train_size), \n",
        "                            'val': (self.val_df, self.val_size),\n",
        "                            'test': (self.test_df, self.test_size)}\n",
        "        self.set_split('train')\n",
        "\n",
        "        # Class weights (for imbalances)\n",
        "        class_counts = df.category.value_counts().to_dict()\n",
        "        def sort_key(item):\n",
        "            return self.vectorizer.category_vocab.lookup_token(item[0])\n",
        "        sorted_counts = sorted(class_counts.items(), key=sort_key)\n",
        "        frequencies = [count for _, count in sorted_counts]\n",
        "        self.class_weights = 1.0 / torch.tensor(frequencies, dtype=torch.float32)\n",
        "\n",
        "    @classmethod\n",
        "    def load_dataset_and_make_vectorizer(cls, df, cutoff):\n",
        "        train_df = df[df.split=='train']\n",
        "        return cls(df, NewsVectorizer.from_dataframe(train_df, cutoff))\n",
        "\n",
        "    @classmethod\n",
        "    def load_dataset_and_load_vectorizer(cls, df, vectorizer_filepath):\n",
        "        vectorizer = cls.load_vectorizer_only(vectorizer_filepath)\n",
        "        return cls(df, vectorizer)\n",
        "\n",
        "    def load_vectorizer_only(vectorizer_filepath):\n",
        "        with open(vectorizer_filepath) as fp:\n",
        "            return NewsVectorizer.from_serializable(json.load(fp))\n",
        "\n",
        "    def save_vectorizer(self, vectorizer_filepath):\n",
        "        with open(vectorizer_filepath, \"w\") as fp:\n",
        "            json.dump(self.vectorizer.to_serializable(), fp)\n",
        "\n",
        "    def set_split(self, split=\"train\"):\n",
        "        self.target_split = split\n",
        "        self.target_df, self.target_size = self.lookup_dict[split]\n",
        "\n",
        "    def __str__(self):\n",
        "        return \"<Dataset(split={0}, size={1})\".format(\n",
        "            self.target_split, self.target_size)\n",
        "\n",
        "    def __len__(self):\n",
        "        return self.target_size\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        row = self.target_df.iloc[index]\n",
        "        title_vector, title_length = self.vectorizer.vectorize(row.title)\n",
        "        category_index = self.vectorizer.category_vocab.lookup_token(row.category)\n",
        "        return {'title': title_vector, 'title_length': title_length, \n",
        "                'category': category_index}\n",
        "\n",
        "    def get_num_batches(self, batch_size):\n",
        "        return len(self) // batch_size\n",
        "\n",
        "    def generate_batches(self, batch_size, collate_fn, shuffle=True, \n",
        "                         drop_last=False, device=\"cpu\"):\n",
        "        dataloader = DataLoader(dataset=self, batch_size=batch_size,\n",
        "                                collate_fn=collate_fn, shuffle=shuffle, \n",
        "                                drop_last=drop_last)\n",
        "        for data_dict in dataloader:\n",
        "            out_data_dict = {}\n",
        "            for name, tensor in data_dict.items():\n",
        "                out_data_dict[name] = data_dict[name].to(device)\n",
        "            yield out_data_dict"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_Dpb6ZHJFMeb",
        "colab_type": "code",
        "outputId": "c24edb33-310c-428f-94fc-9e53446e080f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# Dataset instance\n",
        "dataset = NewsDataset.load_dataset_and_make_vectorizer(df=split_df,\n",
        "                                                       cutoff=args.cutoff)\n",
        "print (dataset) # __str__\n",
        "input_ = dataset[5] # __getitem__\n",
        "print (input_['title'], input_['title_length'], input_['category'])\n",
        "print (dataset.vectorizer.unvectorize(input_['title']))\n",
        "print (dataset.class_weights)"
      ],
      "execution_count": 83,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<Dataset(split=train, size=84000)\n",
            "[ 2 31 32 10 33 13  3] 7 0\n",
            "<BEGIN> software firm to cut jobs <END>\n",
            "tensor([3.3333e-05, 3.3333e-05, 3.3333e-05, 3.3333e-05])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "_IUIqtbvFUAG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Model"
      ]
    },
    {
      "metadata": {
        "id": "xJV5WlDiFVVz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "input → embedding → RNN → FC "
      ]
    },
    {
      "metadata": {
        "id": "rZCzdZZ9FMhm",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch.nn as nn\n",
        "import torch.nn.functional as F"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wbWO4lZcIdqZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def gather_last_relevant_hidden(hiddens, x_lengths):\n",
        "    x_lengths = x_lengths.long().detach().cpu().numpy() - 1\n",
        "    out = []\n",
        "    for batch_index, column_index in enumerate(x_lengths):\n",
        "        out.append(hiddens[batch_index, column_index])\n",
        "    return torch.stack(out)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9TT66Y-UFMcZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class NewsModel(nn.Module):\n",
        "    def __init__(self, embedding_dim, num_embeddings, rnn_hidden_dim, \n",
        "                 hidden_dim, output_dim, num_layers, bidirectional, dropout_p, \n",
        "                 pretrained_embeddings=None, freeze_embeddings=False, \n",
        "                 padding_idx=0):\n",
        "        super(NewsModel, self).__init__()\n",
        "        \n",
        "        if pretrained_embeddings is None:\n",
        "            self.embeddings = nn.Embedding(embedding_dim=embedding_dim,\n",
        "                                          num_embeddings=num_embeddings,\n",
        "                                          padding_idx=padding_idx)\n",
        "        else:\n",
        "            pretrained_embeddings = torch.from_numpy(pretrained_embeddings).float()\n",
        "            self.embeddings = nn.Embedding(embedding_dim=embedding_dim,\n",
        "                                           num_embeddings=num_embeddings,\n",
        "                                           padding_idx=padding_idx,\n",
        "                                           _weight=pretrained_embeddings)\n",
        "        \n",
        "        # Conv weights\n",
        "        self.gru = nn.GRU(input_size=embedding_dim, hidden_size=rnn_hidden_dim, \n",
        "                          num_layers=num_layers, batch_first=True, \n",
        "                          bidirectional=bidirectional)\n",
        "     \n",
        "        # FC weights\n",
        "        self.dropout = nn.Dropout(dropout_p)\n",
        "        self.fc1 = nn.Linear(rnn_hidden_dim, hidden_dim)\n",
        "        self.fc2 = nn.Linear(hidden_dim, output_dim)\n",
        "        \n",
        "        if freeze_embeddings:\n",
        "            self.embeddings.weight.requires_grad = False\n",
        "\n",
        "    def forward(self, x_in, x_lengths, apply_softmax=False):\n",
        "        \n",
        "        # Embed\n",
        "        x_in = self.embeddings(x_in)\n",
        "            \n",
        "        # Feed into RNN\n",
        "        out, h_n = self.gru(x_in)\n",
        "        \n",
        "        # Gather the last relevant hidden state\n",
        "        out = gather_last_relevant_hidden(out, x_lengths)\n",
        "\n",
        "        # FC layers\n",
        "        z = self.dropout(out)\n",
        "        z = self.fc1(z)\n",
        "        z = self.dropout(z)\n",
        "        y_pred = self.fc2(z)\n",
        "\n",
        "        if apply_softmax:\n",
        "            y_pred = F.softmax(y_pred, dim=1)\n",
        "        return y_pred"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jHPYCPd7Fl3M",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Training"
      ]
    },
    {
      "metadata": {
        "id": "D3seBMA7FlcC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch.optim as optim"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "HnRKWLekFlnM",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class Trainer(object):\n",
        "    def __init__(self, dataset, model, model_state_file, save_dir, device, shuffle, \n",
        "               num_epochs, batch_size, learning_rate, early_stopping_criteria):\n",
        "        self.dataset = dataset\n",
        "        self.class_weights = dataset.class_weights.to(device)\n",
        "        self.model = model.to(device)\n",
        "        self.save_dir = save_dir\n",
        "        self.device = device\n",
        "        self.shuffle = shuffle\n",
        "        self.num_epochs = num_epochs\n",
        "        self.batch_size = batch_size\n",
        "        self.loss_func = nn.CrossEntropyLoss(self.class_weights)\n",
        "        self.optimizer = optim.Adam(self.model.parameters(), lr=learning_rate)\n",
        "        self.scheduler = optim.lr_scheduler.ReduceLROnPlateau(\n",
        "            optimizer=self.optimizer, mode='min', factor=0.5, patience=1)\n",
        "        self.train_state = {\n",
        "            'done_training': False,\n",
        "            'stop_early': False, \n",
        "            'early_stopping_step': 0,\n",
        "            'early_stopping_best_val': 1e8,\n",
        "            'early_stopping_criteria': early_stopping_criteria,\n",
        "            'learning_rate': learning_rate,\n",
        "            'epoch_index': 0,\n",
        "            'train_loss': [],\n",
        "            'train_acc': [],\n",
        "            'val_loss': [],\n",
        "            'val_acc': [],\n",
        "            'test_loss': -1,\n",
        "            'test_acc': -1,\n",
        "            'model_filename': model_state_file}\n",
        "    \n",
        "    def update_train_state(self):\n",
        "\n",
        "        # Verbose\n",
        "        print (\"[EPOCH]: {0} | [LR]: {1} | [TRAIN LOSS]: {2:.2f} | [TRAIN ACC]: {3:.1f}% | [VAL LOSS]: {4:.2f} | [VAL ACC]: {5:.1f}%\".format(\n",
        "          self.train_state['epoch_index'], self.train_state['learning_rate'], \n",
        "            self.train_state['train_loss'][-1], self.train_state['train_acc'][-1], \n",
        "            self.train_state['val_loss'][-1], self.train_state['val_acc'][-1]))\n",
        "\n",
        "        # Save one model at least\n",
        "        if self.train_state['epoch_index'] == 0:\n",
        "            torch.save(self.model.state_dict(), self.train_state['model_filename'])\n",
        "            self.train_state['stop_early'] = False\n",
        "\n",
        "        # Save model if performance improved\n",
        "        elif self.train_state['epoch_index'] >= 1:\n",
        "            loss_tm1, loss_t = self.train_state['val_loss'][-2:]\n",
        "\n",
        "            # If loss worsened\n",
        "            if loss_t >= self.train_state['early_stopping_best_val']:\n",
        "                # Update step\n",
        "                self.train_state['early_stopping_step'] += 1\n",
        "\n",
        "            # Loss decreased\n",
        "            else:\n",
        "                # Save the best model\n",
        "                if loss_t < self.train_state['early_stopping_best_val']:\n",
        "                    torch.save(self.model.state_dict(), self.train_state['model_filename'])\n",
        "\n",
        "                # Reset early stopping step\n",
        "                self.train_state['early_stopping_step'] = 0\n",
        "\n",
        "            # Stop early ?\n",
        "            self.train_state['stop_early'] = self.train_state['early_stopping_step'] \\\n",
        "              >= self.train_state['early_stopping_criteria']\n",
        "        return self.train_state\n",
        "  \n",
        "    def compute_accuracy(self, y_pred, y_target):\n",
        "        _, y_pred_indices = y_pred.max(dim=1)\n",
        "        n_correct = torch.eq(y_pred_indices, y_target).sum().item()\n",
        "        return n_correct / len(y_pred_indices) * 100\n",
        "    \n",
        "    def pad_seq(self, seq, length):\n",
        "        vector = np.zeros(length, dtype=np.int64)\n",
        "        vector[:len(seq)] = seq\n",
        "        vector[len(seq):] = self.dataset.vectorizer.title_vocab.mask_index\n",
        "        return vector\n",
        "    \n",
        "    def collate_fn(self, batch):\n",
        "        \n",
        "        # Make a deep copy\n",
        "        batch_copy = copy.deepcopy(batch)\n",
        "        processed_batch = {\"title\": [], \"title_length\": [], \"category\": []}\n",
        "        \n",
        "        # Get max sequence length\n",
        "        get_length = lambda sample: len(sample[\"title\"])\n",
        "        max_seq_length = max(map(get_length, batch))\n",
        "        \n",
        "        # Pad\n",
        "        for i, sample in enumerate(batch_copy):\n",
        "            padded_seq = self.pad_seq(sample[\"title\"], max_seq_length)\n",
        "            processed_batch[\"title\"].append(padded_seq)\n",
        "            processed_batch[\"title_length\"].append(sample[\"title_length\"])\n",
        "            processed_batch[\"category\"].append(sample[\"category\"])\n",
        "            \n",
        "        # Convert to appropriate tensor types\n",
        "        processed_batch[\"title\"] = torch.LongTensor(\n",
        "            processed_batch[\"title\"])\n",
        "        processed_batch[\"title_length\"] = torch.LongTensor(\n",
        "            processed_batch[\"title_length\"])\n",
        "        processed_batch[\"category\"] = torch.LongTensor(\n",
        "            processed_batch[\"category\"])\n",
        "        \n",
        "        return processed_batch   \n",
        "  \n",
        "    def run_train_loop(self):\n",
        "        for epoch_index in range(self.num_epochs):\n",
        "            self.train_state['epoch_index'] = epoch_index\n",
        "      \n",
        "            # Iterate over train dataset\n",
        "\n",
        "            # initialize batch generator, set loss and acc to 0, set train mode on\n",
        "            self.dataset.set_split('train')\n",
        "            batch_generator = self.dataset.generate_batches(\n",
        "                batch_size=self.batch_size, collate_fn=self.collate_fn, \n",
        "                shuffle=self.shuffle, device=self.device)\n",
        "            running_loss = 0.0\n",
        "            running_acc = 0.0\n",
        "            self.model.train()\n",
        "\n",
        "            for batch_index, batch_dict in enumerate(batch_generator):\n",
        "                # zero the gradients\n",
        "                self.optimizer.zero_grad()\n",
        "\n",
        "                # compute the output\n",
        "                y_pred = self.model(batch_dict['title'], batch_dict['title_length'])\n",
        "\n",
        "                # compute the loss\n",
        "                loss = self.loss_func(y_pred, batch_dict['category'])\n",
        "                loss_t = loss.item()\n",
        "                running_loss += (loss_t - running_loss) / (batch_index + 1)\n",
        "\n",
        "                # compute gradients using loss\n",
        "                loss.backward()\n",
        "\n",
        "                # use optimizer to take a gradient step\n",
        "                self.optimizer.step()\n",
        "                \n",
        "                # compute the accuracy\n",
        "                acc_t = self.compute_accuracy(y_pred, batch_dict['category'])\n",
        "                running_acc += (acc_t - running_acc) / (batch_index + 1)\n",
        "\n",
        "            self.train_state['train_loss'].append(running_loss)\n",
        "            self.train_state['train_acc'].append(running_acc)\n",
        "\n",
        "            # Iterate over val dataset\n",
        "\n",
        "            # # initialize batch generator, set loss and acc to 0; set eval mode on\n",
        "            self.dataset.set_split('val')\n",
        "            batch_generator = self.dataset.generate_batches(\n",
        "                batch_size=self.batch_size, collate_fn=self.collate_fn, \n",
        "                shuffle=self.shuffle, device=self.device)\n",
        "            running_loss = 0.\n",
        "            running_acc = 0.\n",
        "            self.model.eval()\n",
        "\n",
        "            for batch_index, batch_dict in enumerate(batch_generator):\n",
        "\n",
        "                # compute the output\n",
        "                y_pred =  self.model(batch_dict['title'], batch_dict['title_length'])\n",
        "\n",
        "                # compute the loss\n",
        "                loss = self.loss_func(y_pred, batch_dict['category'])\n",
        "                loss_t = loss.to(\"cpu\").item()\n",
        "                running_loss += (loss_t - running_loss) / (batch_index + 1)\n",
        "\n",
        "                # compute the accuracy\n",
        "                acc_t = self.compute_accuracy(y_pred, batch_dict['category'])\n",
        "                running_acc += (acc_t - running_acc) / (batch_index + 1)\n",
        "\n",
        "            self.train_state['val_loss'].append(running_loss)\n",
        "            self.train_state['val_acc'].append(running_acc)\n",
        "\n",
        "            self.train_state = self.update_train_state()\n",
        "            self.scheduler.step(self.train_state['val_loss'][-1])\n",
        "            if self.train_state['stop_early']:\n",
        "                break\n",
        "          \n",
        "    def run_test_loop(self):\n",
        "        # initialize batch generator, set loss and acc to 0; set eval mode on\n",
        "        self.dataset.set_split('test')\n",
        "        batch_generator = self.dataset.generate_batches(\n",
        "            batch_size=self.batch_size, collate_fn=self.collate_fn, \n",
        "            shuffle=self.shuffle, device=self.device)\n",
        "        running_loss = 0.0\n",
        "        running_acc = 0.0\n",
        "        self.model.eval()\n",
        "\n",
        "        for batch_index, batch_dict in enumerate(batch_generator):\n",
        "            # compute the output\n",
        "            y_pred =  self.model(batch_dict['title'], batch_dict['title_length'])\n",
        "\n",
        "            # compute the loss\n",
        "            loss = self.loss_func(y_pred, batch_dict['category'])\n",
        "            loss_t = loss.item()\n",
        "            running_loss += (loss_t - running_loss) / (batch_index + 1)\n",
        "\n",
        "            # compute the accuracy\n",
        "            acc_t = self.compute_accuracy(y_pred, batch_dict['category'])\n",
        "            running_acc += (acc_t - running_acc) / (batch_index + 1)\n",
        "\n",
        "        self.train_state['test_loss'] = running_loss\n",
        "        self.train_state['test_acc'] = running_acc\n",
        "    \n",
        "    def plot_performance(self):\n",
        "        # Figure size\n",
        "        plt.figure(figsize=(15,5))\n",
        "\n",
        "        # Plot Loss\n",
        "        plt.subplot(1, 2, 1)\n",
        "        plt.title(\"Loss\")\n",
        "        plt.plot(trainer.train_state[\"train_loss\"], label=\"train\")\n",
        "        plt.plot(trainer.train_state[\"val_loss\"], label=\"val\")\n",
        "        plt.legend(loc='upper right')\n",
        "\n",
        "        # Plot Accuracy\n",
        "        plt.subplot(1, 2, 2)\n",
        "        plt.title(\"Accuracy\")\n",
        "        plt.plot(trainer.train_state[\"train_acc\"], label=\"train\")\n",
        "        plt.plot(trainer.train_state[\"val_acc\"], label=\"val\")\n",
        "        plt.legend(loc='lower right')\n",
        "\n",
        "        # Save figure\n",
        "        plt.savefig(os.path.join(self.save_dir, \"performance.png\"))\n",
        "\n",
        "        # Show plots\n",
        "        plt.show()\n",
        "    \n",
        "    def save_train_state(self):\n",
        "        self.train_state[\"done_training\"] = True\n",
        "        with open(os.path.join(self.save_dir, \"train_state.json\"), \"w\") as fp:\n",
        "            json.dump(self.train_state, fp)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ICkiOaGtFlk-",
        "colab_type": "code",
        "outputId": "3e2b70e0-a724-4953-c803-ac93a8a8b687",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "cell_type": "code",
      "source": [
        "# Initialization\n",
        "dataset = NewsDataset.load_dataset_and_make_vectorizer(df=split_df,\n",
        "                                                       cutoff=args.cutoff)\n",
        "dataset.save_vectorizer(args.vectorizer_file)\n",
        "vectorizer = dataset.vectorizer\n",
        "model = NewsModel(embedding_dim=args.embedding_dim, \n",
        "                  num_embeddings=len(vectorizer.title_vocab), \n",
        "                  rnn_hidden_dim=args.rnn_hidden_dim,\n",
        "                  hidden_dim=args.hidden_dim,\n",
        "                  output_dim=len(vectorizer.category_vocab),\n",
        "                  num_layers=args.num_layers,\n",
        "                  bidirectional=args.bidirectional,\n",
        "                  dropout_p=args.dropout_p, \n",
        "                  pretrained_embeddings=None, \n",
        "                  padding_idx=vectorizer.title_vocab.mask_index)\n",
        "print (model.named_modules)"
      ],
      "execution_count": 89,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<bound method Module.named_modules of NewsModel(\n",
            "  (embeddings): Embedding(3406, 100, padding_idx=0)\n",
            "  (gru): GRU(100, 128, batch_first=True)\n",
            "  (dropout): Dropout(p=0.1)\n",
            "  (fc1): Linear(in_features=128, out_features=100, bias=True)\n",
            "  (fc2): Linear(in_features=100, out_features=4, bias=True)\n",
            ")>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "tuaRZ4DiFlh1",
        "colab_type": "code",
        "outputId": "5a0c146d-cf5d-4d05-ed4e-ece620c1b33e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "cell_type": "code",
      "source": [
        "# Train\n",
        "trainer = Trainer(dataset=dataset, model=model, \n",
        "                  model_state_file=args.model_state_file, \n",
        "                  save_dir=args.save_dir, device=args.device,\n",
        "                  shuffle=args.shuffle, num_epochs=args.num_epochs, \n",
        "                  batch_size=args.batch_size, learning_rate=args.learning_rate, \n",
        "                  early_stopping_criteria=args.early_stopping_criteria)\n",
        "trainer.run_train_loop()"
      ],
      "execution_count": 90,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[EPOCH]: 0 | [LR]: 0.001 | [TRAIN LOSS]: 0.75 | [TRAIN ACC]: 70.1% | [VAL LOSS]: 0.54 | [VAL ACC]: 80.4%\n",
            "[EPOCH]: 1 | [LR]: 0.001 | [TRAIN LOSS]: 0.48 | [TRAIN ACC]: 82.7% | [VAL LOSS]: 0.48 | [VAL ACC]: 82.7%\n",
            "[EPOCH]: 2 | [LR]: 0.001 | [TRAIN LOSS]: 0.41 | [TRAIN ACC]: 85.1% | [VAL LOSS]: 0.47 | [VAL ACC]: 83.1%\n",
            "[EPOCH]: 3 | [LR]: 0.001 | [TRAIN LOSS]: 0.36 | [TRAIN ACC]: 86.6% | [VAL LOSS]: 0.47 | [VAL ACC]: 83.6%\n",
            "[EPOCH]: 4 | [LR]: 0.001 | [TRAIN LOSS]: 0.32 | [TRAIN ACC]: 88.3% | [VAL LOSS]: 0.48 | [VAL ACC]: 83.4%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "mzRJIz88Flfe",
        "colab_type": "code",
        "outputId": "0c47971c-838c-4c04-ee63-0f71c3264c49",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 335
        }
      },
      "cell_type": "code",
      "source": [
        "# Plot performance\n",
        "trainer.plot_performance()"
      ],
      "execution_count": 91,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2gAAAE+CAYAAAD4XjP+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs/Xd0XPWB//8/p6qXUe+yLRe5W+6W\n3MG40vsGQpbk7AlLsr+zYVnzUbIhsLScb5xPgGxI+bBLQrKBADYOmGqwjbtsSzZukrskS1azep2i\n+f0hMdjYuGBJVyO9Huf4MHPv3JmXxkaal97v+74mr9frRURERERERAxnNjqAiIiIiIiIdFFBExER\nERER6SdU0ERERERERPoJFTQREREREZF+QgVNRERERESkn1BBExERERER6SdU0ES+oVGjRlFRUWF0\nDBERkT5xzz33cNNNNxkdQ2TAU0ETERERkUs6cuQIYWFhJCUlUVBQYHQckQFNBU2kh3V0dPDTn/6U\nxYsXs3TpUp577jk8Hg8Af/7zn1m6dClLlizhjjvu4OjRo5fcLiIi0h+sWbOGJUuWsGLFCt5++23f\n9rfffpvFixezePFiHn30UZxO59du37lzJ4sWLfIde+79F198kZ/85CfccccdvPLKK3R2dvLEE0+w\nePFiFi5cyKOPPorL5QKgtraW73//+1x33XXceOONbNmyhY0bN7JixYrzMt92222sX7++t98akR5n\nNTqAyEDzxz/+kYqKCtatW4fb7ea+++7j3Xff5brrruP5559nw4YNhIaG8v7777Nx40YSExMvun3E\niBFGfykiIiJ4PB4+/vhjHn74YSwWC6tWrcLpdFJVVcXPf/5z3n77beLi4vjhD3/In/70J5YsWXLR\n7ePHj7/k62zatIm1a9cSFRXFhx9+yO7du3n33Xfp7Ozk1ltv5b333uPmm29m1apVZGRk8Nvf/pZD\nhw7xj//4j2zevJnq6moKCwvJzMykvLyckpIS5s6d20fvkkjPUUET6WEbN27kwQcfxGq1YrVaufHG\nG9m6dSvLli3DZDLx5ptvsmLFCpYuXQqAy+W66HYREZH+YMuWLYwfP57Q0FAApk+fzoYNG6ivrycr\nK4v4+HgAVq1ahcVi4a233rro9j179lzydSZOnEhUVBQAixcvZsGCBdhsNgDGjx9PaWkp0FXk/vCH\nPwAwZswYPvnkE+x2O4sXL2bdunVkZmayfv16rrvuOux2e8+/ISK9TFMcRXpYbW0tERERvvsRERGc\nPXsWm83GK6+8Qn5+PosXL+Yf/uEfKCoq+trtIiIi/cHq1avZuHEjU6dOZerUqXz00UesWbOGuro6\nwsPDfY8LCAjAarV+7fbLOfdnZ21tLStXrmTx4sUsWbKETz75BK/XC0B9fT1hYWG+x35RHJcvX866\ndesAWL9+PcuWLbu2L1zEICpoIj0sJiaG+vp63/36+npiYmKArt/0vfDCC2zfvp3Zs2fz+OOPX3K7\niIiIkRoaGsjLy2Pnzp3s3r2b3bt3s2vXLvbv34/ZbKaurs732ObmZmpqanA4HBfdbrFYfOdkAzQ2\nNn7t6/7f//t/sVqtvPPOO3zwwQfMmzfPty8yMvK85z99+jQul4tp06bhdrvZsGEDR48eJTs7u6fe\nBpE+pYIm0sPmz5/Pm2++icfjobW1lbVr1zJv3jyKior4l3/5F5xOJ3a7nXHjxmEymb52u4iIiNHW\nrVvHzJkzz5sqaLVamT17Nk6nk/z8fE6fPo3X6+Xxxx/nzTffZN68eRfdHhsbS3V1NWfPnsXj8fDO\nO+987euePXuWkSNHYrfbKSwspKCggNbWVgAWLlzImjVrADh27Bi33XYbHo8Hs9nMsmXL+M///E8W\nLlzomx4p4m90DprINbj//vuxWCy++0899RT3338/paWlLF++HJPJxJIlS3znlaWkpLBixQpsNhsh\nISH89Kc/ZeTIkRfdLiIiYrS3336bBx544ILtixYt4je/+Q1PPvkkDzzwABaLhfHjx/OP//iPBAQE\nfO3222+/nVtuuYWkpCRuvvlmDh8+fNHXffDBB1m5ciWrV69m6tSprFy5kh//+MdMmDCBRx99lJUr\nV7Jw4UJCQkL4xS9+QWBgINA1zfF//ud/NL1R/JrJ+8WEXhERERERP1ZTU8Ott97Kxo0bz/sFqog/\n0RRHERERERkQXnjhBe69916VM/FrKmgiIiIi4tdqamq47rrrqKmp4cEHHzQ6jsg10RRHERERERGR\nfkIjaCIiIiIiIv2ECpqIiIiIiEg/0efL7FdXN13zczgcwdTVtfZAmr7hT3mVtXf4U1bwr7zK2jt6\nKmtsbFgPpBk8BtvPSH/KCv6VV1l7h7L2Hn/K2xNZL/Xz0S9H0KxW/1qZx5/yKmvv8Kes4F95lbV3\n+FNWOZ8//d35U1bwr7zK2juUtff4U97ezuqXBU1ERERERGQgUkETERERERHpJ1TQRERERERE+gkV\nNBERERERkX5CBU1ERERERKSfUEETERERERHpJ1TQRERERERE+gkVNBGRQWTjxk+u6HFPP/005eVl\nvZxGREREvkoFTURkkDhzppz16z+8osf++Mc/JikpuZcTiYiIyFdZjQ5wtTqcHt7ZfIJJwxwE2v0u\nvoiIYX75y59z+PBB5syZxg03LOXMmXJ+9avf8OyzT1JdXUVbWxsPPvhP5OTM4f777+cHP/gRGzZ8\nQktLMyUlxZSVneZf/uURZs3KMfpLERER6VOdXi+llc0cPV3PlLGJOIJ6r4f4XcMpKq3j92/v57op\nKXxr0Uij44iI+I17772f1av/xtChGZSUnOI3v/l/1NXVMn36TJYuXUFZ2Wn+4z8eIydnznnHVVVV\n8otfvMCOHdtYu/YtFTQRERnwvF4v5WdbKSyu43BxHUUldbS0uwE4VdXM95aN7rXX9ruCNmZIFInR\nIWwsKOP6qSnEO4KNjiQictX+9ukxdhVW9ehzTsuM466Fw6/osaNHjwUgLCycw4cP8ve/r8ZkMtPY\n2HDBYydMmARAXFwczc3NPRdYRESkn/B6vVTVt3G4uI7C4joKS+ppbHH69keHB5I1IpbM9EiunzmU\n1ub2XsvidwXNajHz7eWj+fmfdrN60wkeumWc0ZFERPyOzWYD4OOPP6CxsZH/+q//R2NjI9/73v0X\nPNZisfhue73ePssoIiLSm842tHcVspKuUbK6pg7fvohQOzPHxpOZ5mB0uoPYyCDfvpAgmwraV+VM\nSGJoYji7Cqu4obyBjKQIoyOJiFyVuxYOv+LRrp5iNpvxeDznbauvrycxMQmz2cymTZ/icrn6NJOI\niEhfqW/u6B4d6ypk1fVflqzQIBtTR8UyOt1BZrqDhKhgTCaTITn9sqCZTCbuWpDBz/+3gDc2HGfl\nP2QZ9gaKiPiL9PShFBUVkpiYRGRkJADz5y/kscd+xKFDB1i+/Cbi4uL4n//5g8FJRURErl1zm6vr\nHLKSrmmLZ862+vYFBViZNDzGV8iSY0Mw95M+4ZcFDWBUmoNJw2PYe6yGfcfOMmlEjNGRRET6NYfD\nwerV687blpiYxB//+Jrv/g03LAUgNjaM6uomhg37cpRv2LDh/PrXv++bsCIiIleptd3NkdJ637TF\n0qovz5sOsFkYNyyqq5ClOUiPD8Ns7h+F7Kv8tqAB3D4/g33Ha3hz03HGZ0RhMeuybiIiIiIig0GH\n08PR0/W+EbJTFU18caq01WLuLmORjE6PYkhiGFaLf3QFvy5oyTEhzJmQxGf7ytm6v4K5E5OMjiQi\nIgNQS0sLK1eupKGhAZfLxcMPP8zvf//laGJVVRW33nor3//+933bXnzxRd555x3i4+MBuOmmm7jz\nzjv7PLuIyEDhcns4Vtbom7Z4srwRT2dXI7OYTWQkRzA6rWvK4vDkcGxWy2WesX/y64IGcPPsoew4\nWMGazSeYMTqeALt//kWIiEj/tWbNGoYOHcojjzxCZWUlDzzwAB988IFv//e+9z1uvvnmC4779re/\nzX333deXUUVEBgy3p5NTZ5o4XFzL4eI6jpU14vZ0AmAywZCEMDLTHYxOczA8JYJAu99XG2AAFDRH\nWAA3TE/j3W2n+GhXCTfmDDU6koiIDDAOh4OioiIAGhsbcTgcvn3btm1jyJAhJCYmGhVPRGRA6Oz0\nUlzZ5BshO1raQIfry9WHU+NCfeeQjUyNJDjQ76vMRQ2Ir2rpjDQ27S3jvZ0lzJuUTHiI3ehIIiIy\ngCxfvpzVq1ezaNEiGhsb+d3vfufb96c//Ync3NyLHvfBBx/wySefYLfb+clPfkJqampfRRYR6fc6\nvV7Kqls4XFzHyYomPj9WQ1uH27c/MTrYN0I2Ki2SsODB8Rl/QBS0oAArN+UM5S8fH+Gdraf41g0j\njY4kIiIDyNq1a0lKSuLll1+msLCQ3NxcVq9eTWVlJa2traSlpV1wzLx585g5cybTpk1j3bp1PPXU\nU+cVu4txOIKx9sA5E7GxYdf8HH3Fn7KCf+VV1t6hrN+c1+vldFUz+4/X8PnRGvYfr6Gxxenbnxgd\nwpxJyUwYHsP44TFEhQcamPbSevO9HRAFDWDepCQ+3l3Kxr1lXD81hfioYKMjiYj4pTvuuJH33lt3\n+QcOIvn5+cyePRuAzMxMqqqq8Hg8bNq0iZkzZ170mAkTJvhuL1y4kF/84heXfZ26utbLPuZyvrhE\ngj/wp6zgX3mVtXco69Wrrm/rWva+e9piQ/OXhcwRFkD2uAQy0xzkZKVg8nw5ndHT4aK62mVE5Mvq\niff2UgVvwBQ0q8XMHfMy+M3bB3hr03H++dbxRkcSEZEBIj09nX379rF48WLKysoICQnBYrGwf/9+\nFixYcNFjnnrqKZYsWcLUqVPJy8tjxIgRfZxaRKTv1Ta2U1hSR2Fx1/XIzja2+/aFB9uYPjqua9pi\nuoO4yCBM3ReHjo0K7heFsj8YMAUNYMqoWIYlhbO7qJrjZQ1kJEcYHUlEpN948MFv8cwzq0hISKCi\n4gz/5/88QmxsHG1tbbS3t/Ov//ooY8aMMzpmv3T33XeTm5vLfffdh9vt5mc/+xkA1dXVREdH+x5X\nXV3Niy++yJNPPsmdd97J448/jtVqxWQy8dRTTxmUXkSk9zS2OLsLWR2Hi+uorGvz7QsJtDJ5ZGzX\nwh7pDpKig32FTL7egCpoJpOJuxYM57m/5PPGhmOs/NZk/SMQEek2d+4Ctm79jNtvv4vNmzcxd+4C\nMjJGMHfufPbs2cVf/vJHnn76/zM6Zr8UEhLC888/f8H23/72t+fdj42N5cknnwRg1KhRvPbaa32S\nT0Skr7S0uygqqfdNWyyrafHtC7RbmJAR7VtpMTU+FLM+i1+1AVXQAEamRpI1IoaCozXsPVZD1ohY\noyOJiFxg9bF3Kaja36PPmRU3ntuGr/ja/XPnLuDXv/4Vt99+F1u2bOIHP/hXXnvtVf7611dxuVwE\nBvbfk7FFRMQYbR1ujp7+opDVU1LZhLd7n91qZuyQrtGxzHQHQxLCsJjNhuYdCAZcQQO4fV4Ge4/V\n8ObG40zIiNY/FBERYNiwDM6eraaysoKmpiY2b95ITEwc//Ef/0lh4SF+/etfGR1RREQM1uHycKys\ngcLuEbKTZ5ro9HZVMqvFxMjUSN85ZEMTw7FZ9Tm7pw3IgpYUE8LciUls2lvO5s/PMH9SstGRRETO\nc9vwFZcc7eots2bN5ve//w1z5syjvr6OjIyuhSs2bdqA2+2+zNEiIjLQuNydnChvoLB72uKJ8gbc\nnq5CZjaZGJoY5itkGckRBNiu/VIgcmkDsqAB3Dx7KNsPVrB280lmjUkgwK5/TCIi8+Yt4Pvff5BX\nXvkr7e1tPPXU42zYsJ7bb7+L9es/Yt26vxsdUUREepGns5NTZ5ooLOla1OPY6Qac7k4ATEBafFj3\noh6RjEiJJChgwNaFfmvAvuORoQEsmZ7G37ee4sNdJdyUM9ToSCIihhs9eiybNu303f/LX9703Z49\nex4Ay5ffREhICK2tWu5YRMTfdXq9lFY2d51DVlLHkdJ62p1fXm8sOTaE0Wld55CNSoskJNBmYFqB\nAVzQABZPT2NjQRnv7yxh3qRkIkLsRkcSEREREek1nV4v5dUt7CisZvehCopK6mhp/3IKe3xUMDPT\nus4jy0xzEK7Px/3OgC5oQQFWbpo9lD9/dIS/bz3J/TeMMjqSiIiIiEiP6ez0UlLVRFFJPUdKu/6c\nW8iiwwPJGvHltcgcYQEGppUrMaALGsDciUl8vKuUTQXlXD8lhcToEKMjiYiIiIh8I25PJ8UVTRSV\n1lNUUs+xsnraOr6cshgdHsjE4TFMGZNASlQQsZFBBqaVb2LAFzSrxcwd8zP4rzUHWL3pBA/fNt7o\nSCIiIiIiV8Tl9nCivJEjpfUUldZzrKwBp6vTtz/eEcS0zEhGpnb9iYnoKmSxsWFUV+tcYn804Asa\nwOSRsWQkh7PnSDXHyhoYnhxhdCQRERERkQt0uDwcL2vwTVk8Xt6I2/NlIUuKCWFUaiSj0rpWWdSU\nxYFnUBQ0k8nEXQuG8+yf8/nbhmP8n29NxmQyGR1LRERERAa5tg43R083dI+Q1XHqTBOezq7rkJmA\n1LhQRqZFMio1khGpkYQHa1GPgW5QFDSAESmRZI2IoeBoDQVHa5g8MtboSCIiIiIyyLS0u7rKWPcI\nWXFlE96uPobZZCI9IZRRqQ5GpkUyIiVCy94PQldU0J555hn27duHyWQiNzeXCRMmAFBZWcm//du/\n+R5XWlrKI488wo033tg7aa/RHfMz2HfsLG9uPM7E4dFYzGajI4mIiIjIANbY4vSdP1ZUUk9ZdTPd\nfQyL2URGckTXlMXUSDKSI3RhaLl8QcvLy6O4uJjXX3+d48ePk5uby+uvvw5AfHw8r776KgBut5v7\n77+fhQsX9m7ia5AYHcLcSUlsLChj874zzM9KNjqSiIiIiAwgdU0dFJXWcaSkq5SdOdvq22ezmhmV\n1rWYx6g0B8OSwgmwWQxMK/3RZQva9u3buf766wHIyMigoaGB5uZmQkNDz3vcmjVrWLx4MSEh/XsZ\n+5tzhrD9QAVvbznJzLHxBNr1WwoRERER+WZq6tu6RsdK6zlSUk9VfZtvX4DNwtihUV2FLDWSoYnh\n2KyawSWXdtl2UlNTw9ixY333o6KiqK6uvqCgvfHGG/z3f//3ZV/Q4QjGar323xTExoZ94+NuWzCc\nv35UxJYDldy7OPOas1zp6/oLZe0d/pQV/CuvsvYOf8oqItIXvF4vlbWtvumKR0rrONvY4dsfFGBl\nQka0b5QsPT4Mq0WFTK7OVQ8feb84i/EcBQUFDBs27ILSdjF1da2XfczlXOt1HWaPjWfd1pO8teEY\n00bGEBHau8uT+tN1KJS1d/hTVvCvvMraO3oqq0qeiPgzr9dLeU3Ledcgqz2nkIUG2Zg8MtY3QpYa\nF4rZrJXC5dpctqDFxcVRU1Pju19VVUVs7PkrIG7cuJFZs2b1fLpeEhRg5ebZQ3n1wyLWbj3FtxeP\nMjqSiIiIiBiss9PL6epm3wqLRaX1NLe5fPsjwwKYlhnnGyFLignBrEs3SQ+7bEHLycnhxRdf5J57\n7uHgwYPExcVdMFK2f/9+li1b1mshe8OcCYl8vKuUz/aWs2hqConR/fvcORERERHpWZ7OTkoquwpZ\nUUkdR0830Nrh9u2PCg9g1rB436Ie40bGUVPTbGBiGQwuW9AmT57M2LFjueeeezCZTDz++OOsXr2a\nsLAwFi1aBEB1dTXR0dG9HrYnWS1mbp+XwX+t2c9bm07wg9vGGx1JRERERHqR29PJyTONvuuQHS1r\noMPp8e2PjQxk8shY3whZTEQgpnNGyEwaLZM+cEXnoJ17rTOAzMzzF9Z45513ei5RH5o8MobhyRHk\nH6nm6Ol6RqREGh1JRERERHqI0+XhRHlj96IedZwob8Tp7vTtT4wO9p0/NjI1kqjwQAPTinQZ1GvM\nm0wm7lownGf+vIe/bThG7n1T9JsRERERET/V7nRzrKzBN0J28kwjbs+XC9ylxIYwKtXBqLRIRqRG\nEhFiNzCtyMUN6oIGMDwlgikjY9lzpJr8IzVMGRV7+YNERERExHCt7S6Onm7wLXtfXNFEZ/eK4yYT\npMWHMap7hGxEaiShQTaDE4tc3qAvaAC3zRtGwdEa3tx0nInDo3W9ChEREZF+qLnNdc4Ki3WUVjbz\nxfiYxWxiaGIYI9MiGZXqYHhyBMGB+qgr/kf/aoHE6BDmTUpiQ0EZm/eVs2ByitGRRERERAa9huaO\nrtGx0q5SVlbd4ttntZgZ0X3u2Ki0SIYnRRBgtxiYVqRnqKB1u2n2ULYdqGDtlpPMHJtAUIDeGhER\n6dLS0sLKlStpaGjA5XLx8MMP8/vf/57W1laCg4MBWLlyJePGjfMd43K5eOyxxygvL8disfDss8+S\nmppq1Jcg4hdqG9u7lrzvLmWVta2+fXarmdHpXeePjUqNZFhSODarCpkMPGoh3SJC7CydkcbbW07y\nYV4Jt8wZZnQkERHpJ9asWcPQoUN55JFHqKys5IEHHiA2NpZnn32WkSNHXvSYd999l/DwcFatWsWW\nLVtYtWoVv/rVr/o4uUj/VtvYzr6Ttew+WEFRaT01De2+fYF2C+OHRTMyNYJRaQ6GJITpNBQZFFTQ\nznHD9FQ2FJTxYV4p87OSiQwNMDqSiIj0Aw6Hg6KiIgAaGxtxOByXPWb79u3ccsstAGRnZ5Obm9ur\nGUX8RbvTzZ6iarYdqKCwuM53DllIoJVJw2N81yBLiw/FYlYhk8FHBe0cgXYrN88eyp8+LOLvW07y\n7SWZlz9IREQGvOXLl7N69WoWLVpEY2Mjv/vd71i1ahUvvPACdXV1ZGRkkJubS2Dgl9dQqqmpISoq\nCgCz2YzJZMLpdGK3a1lvGXw6vV6KiuvYeqCCPUXVdLi6Lg49IiWC+VNTSYkKJjk2BLMudySigvZV\ncyYm8tGuUj7bd4brp6aSFBNidCQRETHY2rVrSUpK4uWXX6awsJDc3FweeughRo0aRVpaGo8//jh/\n+ctf+O53v/u1z+H1er923xccjmCsPXBOTWxs2DU/R1/xp6zgX3n7Q9ay6mY+3V3Khj2lVNe1ARAf\nFczCqaksmJJKoh9+zuoP7+uV8qes4F95ezOrCtpXWMxm7pyfwYur9/PWpuP88PYJRkcSERGD5efn\nM3v2bAAyMzOpqqpi4cKFWCxdZWrhwoW899575x0TFxdHdXU1mZmZuFwuvF7vZUfP6upaL7n/SsTG\nhlFd3XTNz9MX/Ckr+FdeI7O2tLvIO1zFtv1nOF7eCHSdTzZ7QiI54xIYkRrZNVLm7aS6uknvay/x\np6zgX3l7IuulCp4K2kVMGhHDiJQICo7WcKS0npGpkUZHEhERA6Wnp7Nv3z4WL15MWVkZwcHBfPe7\n3+WFF14gPDycnTt3MmLEiPOOycnJ4YMPPmDOnDls2LCBGTNmGJRepPe5PZ0cOFnLtgMV7D1ag9vT\niQkYO8RB9vhEJo+MJcCmFRdFroQK2kWYTCbuXDCcZ17dwxsbjpF7/xRMmhMtIjJo3X333eTm5nLf\nfffhdrt54oknqKur4zvf+Q5BQUHEx8fzwx/+EICHHnqIl156iWXLlrFt2zbuvfde7HY7zz33nMFf\nhUjPK6lsYtuBCnYcrKCx1QVAYnQwOeMTmTU2AUeYFlwTuVoqaF9jeHIEU0bFsqeomj1F1UzNjDM6\nkoiIGCQkJITnn3/+gu3Lli27YNtLL70E4Lv2mchA09DiZMfBCrYdqKC0qhnoWoHxuskpZI9PYEhC\nmH6xLXINVNAu4fZ5Gew9WsObm44zaUSMrr0hIiIig5LL7WHvsbNs3X+GAydq6fR6sZhNZI2IIXtc\nIhOHR+tzkkgPUUG7hISoYOZNSuLT/DI27S3nuikpRkcSERER6RNer5fj5Y1sO1BB3qFKWjvcAKQn\nhJE9LoEZY+IJD9ZlI0R6mgraZdyUM5StByr4+9aTZI9LIChAb5mIiIgMXGcb2tnWPYWxsrZrZdGI\nUDtLJqWRPS6BlNhQgxOKDGxqG5cRHmJn6Yw03t58kg92lnDr3GFGRxIRERHpUe1ON3uKqtl2oILC\n4jq8gM1qZsaYeLLHJTBmiAOLWVMYRfqCCtoVWDwtjQ35ZXy4q4T5WclakUhERET8XqfXS1FxHVsP\nVLCnqJoOlweAESkR5IxPZOqoOIID9VFRpK/p/7orEGC3cMucofzxgyLWbjnJd5ZmGh1JRERE5Bup\nqG1l24EzbD9QwdnGDgBiIgJZPC6V7HEJxDmCDU4oMripoF2h2RMS+WhXKZs/L2fRtFSSY0KMjiQi\nIiJyRVraXeQdrmLb/jMcL28EINBuYfaERHLGJTAiNRKzlsYX6RdU0K6QxWzmjvkZvPjWft7aeJx/\nuWOC0ZFEREREvpbb08mBk7Vs23+GvcdqcHu8mICxQxxkj09k8shYAmwWo2OKyFeooF2FScNjGJkS\nwd5jNRSV1DEqzWF0JBEREZHzlFQ2sXZbMRt2l9DY6gIgMTqYnPGJzBqboHPpRfo5FbSrYDKZuHPh\ncJ7+0x7+tuE4P/n2FEyaDiAiIiIGa2hxsqN7afzSqmYAQgKtXDc5hezxCQxJCNNnFhE/oYJ2lTKS\nIpiaGcfuwip2F1UzLTPO6EgiIiIyCLncHvYeO8vW/Wc4cKKWTq8Xi9lE1ogYluYMZUhsCFaLlsYX\n8TcqaN/A7fOGUXCkmrc2HSdrRIy++YmIiEif8Hq9HC9vZNuBCvIOVdLa4QYgPSGM7HEJzBgTT3iw\nndjYMKqrmwxOKyLfhAraNxDvCGb+pGQ+yT/Npr3lXDclxehIIiIiMoCdbWhnW/cUxsraVgAiQu0s\nmZRG9rgEUmJDDU4oRuv0dtLiaqXJ2UyTs5nj7V7amt3YLTZsZjsBFvs5t23YLXbMJg0y9EcqaN/Q\njTlD2HrgDGu3nCR7XAJBAXorRUREpOe0O93sKapm24EKCovr8AI2q5kZY+LJGZfA6CEOLGZ9wB6o\nvF4vHZ4OmpwtNLm6Slezs9l32/en+36LqxUv3qt6DavJgs3SXd7MNmwWW/dtOzaLDbu5676te3/X\nbRt2c1fZs3dvt3c/h637tv0t+krYAAAgAElEQVScx6gEXj21im8oPMTO0pnprPnsBO/vLOa2uRlG\nRxIRERE/1+n1UlRcx9YDFewpqqbD5QFgREoEOeMTmToqjuBAfXzzV55OD82ulvOK1blFq9nZfF4h\nc3W6LvucwdYgwuyhxAfHEWYP7fpjCyHO4aC+sQWnx4mz04XT48LZ6ey673F1b3P69rd7Omh0NuPs\ndNLp7eyxr9lqtvpK3JfFzX7BtojSEDxOzil/5zy2uyx23bafd9tmtg64Eqj/w6/BDdNS2ZB/mo/y\nSlmQlaJla0VEROQbqahtZduBM2w/UMHZxg4AYiICWTwulexxCcQ5gg1OKBfj9Xpp97TT5Gym8YIR\nrhZf6fpiX4u79bLPaTVZCLOHkRgSR6g9lDBbKOH2MELtIYTZQr8sYfZQQm0hWM0X/zh/Lechejo9\ndHic3YXOdU7Jc+LqdHXtu6DwOXF5uvd1unB5nOfc7jq2zd1Oo6eJDo/zqkf7LsVmtnYXt4uXua+O\nDH45Ymi/6CjhF1NB7d2PtfZxCVRBuwYBNgu3zBnGK+8XsnbLCb6zdLTRkURERMRPtLS7yDtcxbb9\nZzhe3ghAoN3C7AmJ5IxLYERqJGYtjd/nXJ3uc4pWi+92o7MJ14kOahrrz5ty6PZ6LvucIbZgwgLC\nSApNOGeUK7SrgNlDCbeHEtpdvgItAYZfEsFithBsDiKYoF55fq/Xi9vrwXVO8QsJt1F5th5nd8lz\nnTvyd05Z7OgugueWxnPLYqurjfrOBpweVw+XQJtvGufyzIXMip7ZY8/9VSpo1yhnfAIf7Spl8+dn\nWDQ1lWSdpCsiIiJfw+3p5MDJWrbtP8PeYzW4PV5MwNihUeSMSyBrZCwBNovRMQeUTm8nbe72C87Z\nOn9a4Zfb29ztl31Om9lGuD2U5LCkC0a1zr0fagsl1BaMxay/03OZTCZsJis2s5UvxoZjHWGEuCN7\n7DW8Xi/uTvcFUzm/WvguNkr4tY/t3u7udPdYzotRQbtGFrOZO+Zn8MKbn/PmxuP8/+6caHQkERER\n6WdKKpvYdqCCHQcraGztOq8oMTqYnPGJzBqboNMkrpLT4+oawXJduFhGk7OFJmfTl+XL1XLZc6pM\nmAi1heAIiCQt7KsjXF9OLRySkICz2USAxd5HX6l8UyaTCZula3pjiK1npwj39mUsVNB6wMSMaEal\nRrLv+FmKSuoYleYwOpKIiIgYrKHFyY7upfFLq5oBCAm0ct3kFLLHJzAkIczwqWz9xblLxH9Rus4/\np6vlvBGvdk/HZZ8z0BJAqD2U9MCo7hGtL4pWWNftc6YVhtiCr+gco9jQMKrbdH056V0qaD3AZDJx\n54LhPPWn3fxtwzF+8u2p+oYrIiIyCLncHvYeO8vW/Wc4cKKWTq8Xi9lE1ogYssclMnF4NFbLwFpx\nDroKVru7gzZ3O+2edlpdbbR72mlzn/unrWv/V7d52mjsaL7s+UJmk5kwWwjRQVFdi2bYQn1F66vT\nCsPsodgttj766kV6lgpaDxmWFM60zDh2FVaxq7CK6aPjjY4kIiIifcDr9XK8vJFt+8+Qd7iK1o6u\n81PSE8LIGZfA9DHxhAf33ylxX1xv66tlqt3dTusXhcpzfsk6v2i1XdGI1leZTWaCrIGEB4QSExjd\nVa6+UrbOvR1kDRxwy6mLXIwKWg+6fd4w8o9U89am40weGTsgf0MmIiIiXc42tLOtewpjZW3X8ukR\noXaWTEojZ1xCnywc5vV6cXa6fIWqrbtU2dugsrbOV6RavyhVnravFLGu/Ve72p0JE0HWQIKsgUQH\nRfluB1mDuv5rCSTwq9u6/wRaAwm2BmEz2zCZTL1+Po+Iv1FB60FxjmAWZCWzfs9pNhSUsWhqqtGR\nREREpAe1O93sKapm24EKCovr8AI2q5kZY+LJGZfA6CEOLOYr/wWty+PqGp1ytZ0zSvV10wHbzyti\nXzz+m1xUONDSVZYcAREEhcT7itNXy9SXRev87QH9YCl4kYFKBa2HrcgZwtYDZ3hn6ylyxiUSHKi3\nWERExN8VldTx5/VH2bqvnA5X13WvhqeEMnVsFJlDwsDiotVdy/6a8vPK09eed9W97UquofVVARY7\nQdYgwuxhxAXFnl+orEEEWgOJjYzA0246bwQr0BJIsK2rXGmqoEj/pfbQw8KD7Sydkc7qz07w/s5i\nbp+XYXQkERG5Ri0tLaxcuZKGhgZcLhcPP/wwsbGxPPnkk5jNZsLDw1m1ahVBQV9e1HX16tU8//zz\npKWlAZCdnc1DDz1k1Jcg31CTs5k3925m5+kDmKxOLGM8RAR00mlyUuZ1U1YHa+uu/PlsZhtB1kBC\nbMHEBEV/OXJluXA64MWmCAZaAq7omlqaNijiv1TQesGiaal8mn+aj3aVsiArmdjYMKMjiYjINViz\nZg1Dhw7lkUceobKykgceeICYmBgee+wxJkyYwM9//nNWr17Nt771rfOOW7ZsGStXrjQotXxTTo+T\nz2sOsauigEO1RXR6O7FEgsVkIdgaRJA1mEBr1OULleX8c64CrQFYzfroJSKXpu8SvSDAZuHWOcP4\nn/cLeXvLSVZmxBodSUREroHD4aCoqAiAxsZGHA4Hv/3tbwkN7VoEIioqivr6eiMjyjXq9HZSVHeM\nXRUF7K3eT4fHCYDDEkfliSiyU7L493+Yq1EpEel1Kmi9JGd8Ih/tKmXr/jMUn2kk2KoTaUVE/NXy\n5ctZvXo1ixYtorGxkd/97ne+ctba2sratWt5/vnnLzguLy+P7373u7jdblauXMmYMWP6Orpcgtfr\n5XRzOXkV+eyp3EuDs6t8RQc6WJAym6zYSTz/5+OYW13ccof+7kSkb6ig9RKz2cSdCzL41Ruf88q6\nQ/zzzWONjiQiIt/Q2rVrSUpK4uWXX6awsJDc3FxWr15Na2srDz30EA8++CAZGeefczxx4kSioqKY\nP38+BQUFrFy5knfeeeeSr+NwBGO1Xv78osvxp6n1RmStajnLluI8NhfnUdZYAUCIPZhFGXOYkz6D\nUTHDMJlMvL/9FGcbO7hpzjBGDosxLO83pay9Q1l7jz/l7c2sKmi9aPywaDLTItl9uJLDk5IYne4w\nOpKIiHwD+fn5zJ49G4DMzEyqqqpwOp388z//MytWrOC222674JiMjAxfacvKyqK2thaPx4PF8vUF\nrK6u9Zqz+tPiEH2ZtcXVSn7V5+yqyOd4wykArGYrWXETmB6fxZjoUb7zw2pqmnG5O3nto0JsVjPz\nJyZSXd2k97aXKGvv8Kes4F95eyLrpQqeClovMplM3LlgOP/5x928seEYP3lgKmZdM0RExO+kp6ez\nb98+Fi9eTFlZGSEhIbz88stMnz6dO++886LH/OEPfyAxMZEVK1Zw5MgRoqKiLlnOpOe5PC72nz3M\nrooCDp4txOP1YMLESMdwpsVnkRU3jiBr0EWP3fJ5ObWNHdwwLZXI0IA+Ti4ig5kKWi8bmhjO3EnJ\nfLa3jF2Hq5gxJt7oSCIicpXuvvtucnNzue+++3C73fzsZz/j0UcfJSUlhe3btwMwY8YMfvCDH/DQ\nQw/x0ksvceONN/Loo4/y2muv4Xa7efrppw3+KgaHTm8nx+pPkFdRQEHVfto97QAkhyYyLT6LqfGT\ncARGXvI5XO5O3t1ejN1qZunM9L6ILSLio4LWB+5fNpqtn5fz1qbjTB4Zi82qi0OKiPiTkJCQCxYB\n2bJly0Uf+9JLLwGQkJDAq6++2uvZpEtZ8xl2VRSwq7KA+o4GABwBkcxJnsm0hCySQxOv+Lk+21dO\nXVMHi6enEhFi763IIiIXpYLWBxKiQ1gwOZn1u0+zsaCMRdNSjY4kIiLi9+ra69lduZe8inzKW7oW\n+wiyBpKdOJ3pCVlkRA7FbLq6X4q63B7e29E9ejZDo2ci0veuqKA988wz7Nu3D5PJRG5uLhMmTPDt\nO3PmDD/60Y9wuVyMGTOGJ598stfC+rMbs4ewdf8Z3tl2ipzxCQQH2oyOJCIi4ndaXW3srd5PXkU+\nx+pP4sWL1WRhYuw4psdnMTY6E5vlm/+M/WzfGeqaOlgyPY1wjZ6JiAEuW9Dy8vIoLi7m9ddf5/jx\n4+Tm5vL666/79j/33HM8+OCDLFq0iCeeeILy8nKSkpJ6NbQ/Cgu2s2xmOm9tOsF7O0q4Y37G5Q8S\nERERXJ1uDp4tZFdFAQfOHsbd6QZgeORQpsdPJituPMG24Gt/HbeHddtPYbeZWTIj7ZqfT0Tkm7hs\nQdu+fTvXX3890LVkcENDA83NzYSGhtLZ2cmePXv45S9/CcDjjz/eu2n93KKpqXyaX8bHu0tZODmZ\nqPBAoyOJiIj0S53eTk40FJNXkU9+1ee0udsASAiJZ0b8ZKbETyI6qGcvX7Nxbzn1zU6WztDomYgY\n57IFraamhrFjv7zIclRUFNXV1YSGhlJbW0tISAjPPvssBw8eZOrUqTzyyCO9Gtif2W0Wbp0zjP9+\n7zBvbz7Jg8tHGx1JRESkXznTUkleRT67Kgqo66gHIMIeTnbqNKYlTCYlNBFTL1yyxuny8N72YgJs\nFo2eiYihrnqREK/Xe97tyspKvv3tb5OcnMw//dM/sXHjRubPn/+1xzscwVit134dGH+60jh8mfem\nBaF8WlDG1gNnuHtxJkMSww1OdiF/em+Vtff4U15l7R3+lFX8W31HA7sr97KrooDTzeUABFoCmJkw\nlWkJWYx0ZFz1Yh9Xa+PechpanCybmU5YsEbPRMQ4ly1ocXFx1NTU+O5XVVURGxsLgMPhICkpibS0\nrt80zZo1i6NHj16yoNXVtV5jZP+60jhcmPeW2UP51Rv7+P3qz/nXuyYamOxC/vTeKmvv8ae8yto7\neiqrSp58nTZ3OxtPHuCTo9s4UnccL17MJjPjY0YzLX4y42PGYL+GxT6uRoera+XGALtGz0TEeJct\naDk5Obz44ovcc889HDx4kLi4OEJDQ7sOtlpJTU3l1KlTDBkyhIMHD7J8+fJeD+3vxg+LYnS6g/0n\nznLoVC1jhkQZHUlERKTXuTvdHK49Ql5FPvtrDuHqXuxjWEQ60+InMzluAqH2kD7PtamgjMYWJ8tn\npRMapFWWRcRYly1okydPZuzYsdxzzz2YTCYef/xxVq9eTVhYGIsWLSI3N5fHHnsMr9fLyJEjWbhw\nYV/k9msmk4k7F2Tw5Cu7eWPDcf7jOw7MvTCfXkRExGher5eTjcXsqihgT9U+WlxdM2nig2OZP2wm\nY8LGEBMUbVi+DpeH93aWEGi3sHi6Rs9ExHhXdA7av/3bv513PzMz03c7PT2dv/71rz2bahAYkhDO\njDHx7DxUSd7hSmaOSTA6koiISI+pbKliV2UBuyoKqGmvBSDMHsqC1NlMi88iLSyFuLhww6f9bsjv\nGj1bka3RMxHpH656kRDpObfNHcbuwipWbzrBlJFx2Ky9ewK0iIhIb2p0NrGnch95FfmUNJ0GwG6x\nMz1hMtPisxjlGI7FfO0LhfWUDqeH93cWExRg4YZpGj0Tkf5BBc1AsZFBXDclhY92lbIh/zQ3aGqF\niIj4mXZ3B5/XHCSvIp/C2qO+xT7GRI9ievxkJsSOJcDSP1dF/LTgNE2tLm7MHqLRMxHpN1TQDLYi\newibPz/DO9tOkTMhkZBA/YAQEZH+zdPpobDuKHkV+XxefRBnpwuA9PBUpsdPZkr8RMLsoQanvLR2\np5v3d5QQFGDlhumpRscREfFRQTNYaJCNFbPSeWPjcd7bXsydC4YbHUlEROQCXq+X4qZS8ioKyK/c\nR5OrGYCYoGimx2cxLSGLuOBYg1NeuU/zy2huc3FTzhD9clRE+hUVtH7guikprN9zmo93n2bh5BSi\nIwKNjiQiIgJAVWsNuyoL2F1RQFVb13VRQ20hzEvJZlr8ZIaEp2Lys5WI2zrcfLCze/RsmkbPRKR/\nUUHrB+w2C7fNHcbL6w7z9uYTfHfFGKMjiYjIINbkbGZP1T52VxRwsrEEAJvZxtT4SUyLz2J01Mh+\ntdjH1fo0/zTNbS5umT2UYI2eiUg/o4LWT8wam8CHeaVsO1DBommppMWHGR1JREQGEafHyefVB9lV\nWcCh2iN0ejsxYWJ01EimxWcxMXYsgVb/n+HxxehZcICV66dq9ExE+h8VtH7CbDZx14IMfvm3fby5\n8Tg/unuS0ZFERGSA6/R2UlR3jF0VBeyt3k+HxwlAWlgy0+KzmBI/iYiAcINT9qxP9pympd3NLXOG\nEhyoj0Ei0v/oO1M/MnZoFKPTHRw4WcvBU7WMHRJldCQRERlgvF4vpc1l7KooYHflXhqdXReKjg50\nsCBlNtMSskgIiTc4Ze9o63DzYV4JIYFWFmn0TET6KRW0fsRkMnHXguE88cou3thwjNHfmYbZz068\nFhGR/qmmrZbdlQXkVRRQ2VoFQIg1mNnJM5keP5lhEel+t9jH1Vq/u5SWdje3zh1GUIA+AolI/6Tv\nTv1MekIYM8fGs+NgJTsPVTJrbILRkURExE81u1ooqPqcvIoCTjScAsBqtpIVN4Hp8VmMiR6F1Tw4\nPgq0trv5MK+UkEAr109JMTqOiMjXGhzflf3MbXOGsbuwitWbTjB1VCw2q/+ulCUiIn3L6XaSX/U5\neRX5HDpbhMfrwYSJkY7hTI/PYlLcOIKsQUbH7HPrd5fS2uHm9nkaPROR/k3fofqhmMggrpuSwod5\npXyyp4wlM9KMjiQiIn5gQ+kW1m3+iDZXOwDJoYlMT5jMlLiJOAIjDU5nnNZ2Fx/uKiU0yMbCyRo9\nE5H+TQWtn1o+awib951h3fZTzJmYSIiu0yIiIpdR3lxBmD2EOUmzmBafRVKopskDfLz7NG0dbu6Y\nn6HRMxHp9/Rdqp8KDbKxPDudNzYcZ932Yu5aMNzoSCIi0s99a/QdxMaGUV3dZHSUfqO13cVHvtGz\nZKPjiIhclgpaP3b9lBQ+3XOa9btPs3ByMjERg++cARGR/qClpYWVK1fS0NCAy+Xi4YcfJjY2lp/9\n7GcAjBo1iieeeOK8Y1wuF4899hjl5eVYLBaeffZZUlO1tHtf+2hXKW0dbu6cn0GgXR97RKT/Mxsd\nQL6ezWrh1rnDcHs6WfPZSaPjiIgMWmvWrGHo0KG8+uqrPP/88zz99NM8/fTT5Obm8tprr9Hc3Mym\nTZvOO+bdd98lPDycv/71r3z/+99n1apVBqUfvFraXXy8u5SwYJ17JiL+QwWtn5s5NoHUuFB2HKyg\npFJTVkREjOBwOKivrwegsbGRyMhIysrKmDBhAgALFixg+/bt5x2zfft2Fi1aBEB2djb5+fl9G1r4\nMK+Utg4PS2ekE2DXisgi4h9U0Po5s8nEnQsy8AJvbDxudBwRkUFp+fLllJeXs2jRIu677z7+/d//\nnfDwcN/+6OhoqqurzzumpqaGqKgoAMxmMyaTCafT2ae5B7PmNhfrd5cSHmxjQZbOPRMR/6HJ2H5g\n3NBoxg5xcPBkLQdOnmXc0GijI4mIDCpr164lKSmJl19+mcLCQh5++GHCwsJ8+71e72Wf40oe43AE\nY+2Ba1/GxoZd/kH9RG9lff+9Q7Q7PXxrSSYpyT13iQG9t71DWXuHP2UF/8rbm1lV0PzEHfOHc+iV\nXbyx4ThjhkRhNpmMjiQiMmjk5+cze/ZsADIzM+no6MDtdvv2V1ZWEhcXd94xcXFxVFdXk5mZicvl\nwuv1YrfbL/k6dXWt15zVn1Zx7K2szW0u/r75BOEhdqaOiOmx19B72zuUtXf4U1bwr7w9kfVSBU9T\nHP1EekIYM8cmUFrVzI6DFUbHEREZVNLT09m3bx8AZWVlhISEkJGRwe7duwH46KOPmDNnznnH5OTk\n8MEHHwCwYcMGZsyY0behB7EP80rocHpYNjOdAJvOPRMR/6IRND9y69yh7CqsZM1nJ5iWGYetB6bB\niIjI5d19993k5uZy33334Xa7+dnPfkZsbCw//elP6ezsZOLEiWRnZwPw0EMP8dJLL7Fs2TK2bdvG\nvffei91u57nnnjP4qxgcmlqdrN9zmogQO/MnJRkdR0Tkqqmg+ZGYiCCun5LKB3klfLKnjCUz0oyO\nJCIyKISEhPD8889fsP1///d/L9j20ksvAfiufSZ964Pu0bPb5gzDrtEzEfFDmuLoZ5ZnpxMSaOXd\nbadobnMZHUdERKTfaGx18umeMiJC7czT6JmI+CkVND8TEmhj+awhtHa4Wbf9lNFxRERE+o0PdpbQ\n4fKwfGa6Rs9ExG/5XUGr72jgNzv/xI4zu3F6BucI0nVTkokOD+CTPaepqW8zOo6IiIjhGlucfJp/\nGkdYgEbPRMSv+WVB+6x4J68e/hu5W5/ijSNrOdNSaXSsPmWzWrhtbgZuj5c1m08YHUdERMRw7+8s\nxunqZNnMdC2iJSJ+ze8K2pDwNF5Y9gSL0xdiM1vZeHorT+1cxS/3/IadZ/YMmlG1GWPjSYsLZfvB\nSoor/OOaESIiIr2hocXJhvwyHGEBzJ2o0TMR8W9+V9AA4kJjuCljCU9l5/K9cfeT6RjB8YZT/Onw\n6/x461O8efTvVLRUGR2zV5lNJu5cOByAv204htfrNTiRiIiIMd7fUYzT3cmKWenYrH750UZExMev\nl9m3mC1kxY0nK248NW1n2Vqex/byXWwo3cKG0i0MjxzK7KSZTIobj83s11/qRY0dEsW4oVEcOFnL\nwZO1jBsWbXQkERGRPtXQ3MHGgjKiwgOYPUGjZyLi/wZMa4kJiubmjKUsH7qIz2sOsaVsB0V1xzhW\nf5KQo2uZmTCVnKTpxIfEGR21R90xP4ODJ2t5Y+NxxgyJwmw2GR1JRESkz7y3owSnu5Pls4Zo9ExE\nBoQBU9C+YDVbmRw3gclxE6hqrWZreR47zuzmk9LP+KT0M0ZEDmN28kwmxo4bEKNqafFhzBqXwLYD\nFWw/WEHO+ESjI4mIiPSJ+uYONu4tIzo8gDkT9PNPRAYG/28olxAXHMutw5ezYthi9lUfYGvZTo7U\nH+do/QlCbSHMTOwaVYsLjjU66jW5dc4w8g5XsWbzCaZlxunaLyIiMii8t70Yl7uT5dlDsFo0eiYi\nA8OALmhfsJmtTI2fxNT4SVS2VrO1bCc7KnazvmQT60s2MdIxnNlJM5gYOxarH46qRUcEsmhqCu/v\nLOGTPadZOjPd6EgiIiK9qq6pg417y4kOD2S2Zo+IyADif23kGsUHx3LbiBXcOGwxe6sPsKV8B0fq\njnGk7hhhttDuUbUZxAb714Iby2al89m+ct7dXsyciUmEBtmMjiQiItJr3ttRjNvTyY05Gj0TkYFl\n0BW0L9gsNqYlZDEtIYuKlkq2luex88wePi7ZyMclG8l0jGB28kwmxIzBYu7/UwZDAm2syB7C658e\n491tp7jnuhFGRxIREekVdU0dbNpbTkxEINnjEoyOIyLSowZtQTtXQkg8t4+4kZuGLaGgej9bynZQ\nWHeUwrqjhNlDmZU4jZykGcQERRkd9ZIWTk7hkz2n+TT/NNdNSSE2MsjoSCIiIj1u3fZTuD2drNC5\nZyIyAOm72jlsFhvTEybzoyn/zI+n/4j5KTm4Oz18VLyBn23/Ob/e+//YW30AT6fH6KgXZbOauW3u\nMNweL2s+O2F0HBERkR5X29jOZ/vKiY3U6JmIDEwaQfsaSaEJ3DnyZm7OWEZB1edsKd/B4dojHK49\nQoQ9jFlJ08lOnE50kMPoqOeZPiaeD/NK2XGokhumpzIkIdzoSCIiIj1m3fZi3B6vRs9EZMDSd7bL\nsFtszEicwiNTHiZ3+r8yLyUbZ6eLD059wuPbn+O/9r3MvuqD/WZUzWwyceeCDADe2HAcr9drcCIR\nEZGecbaha/QsLjJIo2ciMmBpBO0qJIcmctfIW7glYxl7KvexpXwnh84WcehsEZEBEd3nqk3HERhp\naM4xQ6IYNyyKAydqOXCylvHD/GtFShERkYtZt/0Unk4vN+YMwWLW75hFZGBSQfsG7BY7s5KmMStp\nGqebytlavpO8inzeP7WeD059wtjoTGYnz2BsdCZmkzE/QO6cP5yDJ/J4Y8Mxxg6Jwmw2GZJDRESk\nJ9Q0tLH58zPEO4KYOTbe6DgiIr1GBe0apYQlcfeoW7ll+HL2VO5lS9lODpw9zIGzh4kMiCA7aTo3\nhiygr9/q1LhQsscnsHV/BdsOVDB7gi7iKSIi/mvd9mKNnonIoKCC1kMCLHayk6aTnTSd0qYytpTt\nYFdlAe+d/Jj3T61nXPRoZifNYEz0qD4bVbt1zjDyDlexZvMJpo+Ow27r/9dzExER+aqa+ja2fH6G\n+KhgZozR6JmIDGwqaL0gNSyZezNv59bhK9hTuZcdVbvYX3OI/TWHcAREkpM0g1lJU4kMiOjVHFHh\ngVw/NYX3d5Swfs9pls1M79XXExER6Q3vbu869+ymbI2eicjAp4LWiwKtAeQkz+CWSdez5/hhtpTv\nYFflXt49+SHvnfqY8dGjyUmeyeioEb02qrZ8Zjqf7S1n3fZTzJmQSFiwvVdeR0REpDdU17exdX8F\nCRo9E5FBQgWtj6SFp/AP4Xdw6/AV7K4sYEvZTvbVHGRfzUGiAx1kJ81gVuI0IgLCevR1gwNt3Jgz\nlNc+Ocq724q59/oRPfr8IiIivemdbd2jZzlDtOCViAwKV1TQnnnmGfbt24fJZCI3N5cJEyb49i1c\nuJCEhAQslq7zm37xi18QH6/fcH2dIGsgc5JnMTtpJiVNp9lStoPdlXt558QHrDv5ERNixjA7aSaj\noob32Kjagqxk1u8u5dP801w3NYW4yKAeeV4REZHeVFXXyrb9FSRGBzN9tD5biMjgcNmClpeXR3Fx\nMa+//jrHjx8nNzeX119//f/f3p2HR1nf+/9/ziSZrJNlkpkkZCcJWyIQ9n1RQETciwdPbeu39rQV\nPb1Ov7aXltqLc/y51H7BS2utbfFqT+vPU63IVxEXUETKmrBDCJCFANlIJnvClm2+fySMBEiCkGRm\nktfjurjIPXPPzWs+Iu3JBDcAACAASURBVPe8531/7k+nfVavXk1gYGCfhRyIDAYDCcFxJATHcX/q\nYnaf2c+20kwO2LM5YM8mws/C9CGTmTJkAsGmm+uq+XgbuX/2UP60Lof/+88T/OjutF56FyIig8N7\n773HunXrnNsHDx5kzJgxzu2Kigruu+8+fvzjHzsfe+211/joo4+cX1refffdLFmypP9CDwAf7ThJ\nm8PB3dOT1D0TkUGjxwJt586dzJs3D4Dk5GTq6upobGwkKCioz8MNFv7e/syKncbMmKmcrD/NtpJM\n9lYc5MMTn7K+cCOjrWnMGDKZYWHJN9xVmzQykg1ZRWTmlLNgYhxJ0cG9/C5ERAauJUuWOIurrKws\nPv30U1asWOF8/gc/+AH33HPPVa/77ne/y8MPP9xvOQeS8ppz7MwuZ0hEIBNH2FwdR0Sk3/T4ab+y\nspKwsDDntsViwW63d9pnxYoVPPTQQ6xcuRKHw9H7KQcJg8FAUkgC3xn1IC9Mf4Ylw+4hMsDK/opD\nvHZgNc/u+j98fuorGpoav/GxjQYDD85NAeC9zfn67yQicoNef/11li1b5tzesWMHiYmJREdrvcne\ntH77pe6Z5p6JyODyjW8ScuUH+5/85CfMnDmTkJAQHn/8cTZs2MDChQu7fH1YWADe3je/HpfV2rs3\n0+hr3zyvmYQhC/nW2NvJrTrB5wVb2Vm0jw8KPuGjwg1MjhnLvOSZpNmGYTBc34nLajWz+UApe46W\nc7rqPBO6uJ7fk8ZWWfuOJ+VV1r7hSVn7y6FDh4iOjsZqtTof+9vf/sby5cuvuf9nn33Gpk2bMJlM\nPPPMM8TFxfVXVI9WXn2OHUfOEBMRyAR1z0RkkOmxQLPZbFRWVjq3KyoqOp2Y7r33XufPs2bNIjc3\nt9sCrabm3I1mdbJazdjtDTd9nP5ys3kt2PiXoQ+wOO4Oss7sY1vJLnYU7WVH0V5sARHtc9WiJhBk\n6nke4N1TE9h7tJw3PzxMnMX/qm8lPWlslbXveFJeZe0bvZV1oBV5a9as4b777nNul5eXc+7cOeLj\n46/ad/bs2UyZMoWJEyfy8ccf89xzz/HHP/6x2+MPxi8xr5X1rc9zcTjg4UUjibS51yX5nj627kpZ\n+4YnZQXPytuXWXss0KZPn85rr73G0qVLOXLkCDabzTn/rKGhgf/4j//gjTfewGQysXv3bm6//fY+\nCzvYBfoEMDduBnNip1NQd5JtJZnstx/i/+Z/zEcFnzHWdgszhkwmJXRol121WFsQ02+JZtvhMrZn\nlzFz9JB+fhciIp4rMzOTZ555xrm9ZcsWpkyZcs19r7zj8cqVK3s8/mD7EvNaWcuqzvLVvmJirYGk\nRrvXe/H0sXVXyto3PCkreFbe3sjaXYHXY4E2btw40tLSWLp0KQaDgRUrVrB27VrMZjPz589n1qxZ\n/Mu//Au+vr6MGjWq2+6Z9A6DwUBKaBIpoUl8q/kussr2sq00kz3lB9hTfoDIABszhkxicvQEAn0C\nrnr9vTOTyDxazgdbC5k0MhJfn5v/tlZEZKArLy8nMDAQk8nkfOzw4cPMnTv3mvs/99xzLFy4kAkT\nJpCVlUVqqtahvB4f7TiJw0H7nRuv8xJ+EZGB5LrmoP3sZz/rtD1ixAjnz9/73vf43ve+17up5LoF\n+QRya/ws5sbNJL/2RPut+isO837+ej488RkZ1tHMiJlMckiis6tmCfZjwcQ4Pt55ii/2FHHn1ETX\nvgkREQ9gt9uxWCxXPRYeHt5p+7XXXuPZZ59lyZIlrFixAm9vbwwGA88991x/R/Y4ZVVnycwpJ9Ya\nxLjh1p5fICIyAH3jm4SIezIYDKSGJZMalkxj6ll2ndnD9tJMdpfvY3f5PqICI5kxZDKTo8YR4BPA\nHZMT2HKglE92nWLWmCGYA0w9/yEiIoNYeno6b775ZqfH/vCHP3TatlqtPPvsswAMHz6cd955p9/y\nDQTrtrd3z+6Zoe6ZiAxeKtAGoCBTIPPiZ3Nb3CzyagvYVtK+APaavHV8WPAJ42xjmBEzmcVTE3jn\ny3w+2nGSf503zNWxRURkECutPEtWTjnxtiDGDYtwdRwREZdRgTaAGQwGhoWlMCwshYamRnaV7WFb\naSaZZ/aSeWYv0YGRhCRGsvlgC/PGx2ILu3q+moiISH9Yt70QB+3ds+tdPkZEZCBSgTZImE1BzE+Y\nw23xs8itKWBbaSYH7dm02crxCTfy26xCHp2ykIiINFdHFRGRQabE3sjuoxXERwYxNlXdMxEZ3FSg\nDTJGg5ERllRGWFKpb2pgR+luPsndSo1PASv3vo45O4hgHzMhvsGEmoIJ8e34ddnPZp8gvIy686PI\nQNHmaONc83nqmxpobG6koamRsItBJJmS1cmQfrFu+0l1z0REOqhAG8SCTWYWJt5KAmNZ9ckXhCWU\nE2S+iP1cFSWNZV2+zoCBYFPQNYu39p9DCPUNJtAnAKPB2I/vSEQuaWptpqGpgYaOgquh6Wyn7cam\ns9R3bDc2ncWB46pj/OeUp7AGhF/j6CK9p9jeyJ5jFSREmRmbou6ZiIgKNGFkooV063AOHY7g8Ucn\nk2gN5ELLBeou1lPXVE/txXrnz3UXv/5Vdrac0w0lXR7XaDBeo3gLvqo7F+Dtr29MRXrQ5mjjbPO5\n9uKquZH6pkuFViMNHduNHY81NDdysbWpx2P6eflhNgViDYnAbArC7BPY/rvJzLDoeCKMlh6PIXKz\n1m3T3DMRkcupQBMAvjUnmcMnqvjNW3uYcUs08ybEEhlmIzLQ1uVrHA4H51vOtxdwlxdvl/1ce7Ge\nooYSTtaf7vI43kbvLou3yx/38/LVyVsGlKbWJmdB1dB02a9rbHfV5bqc0WDE7BOI1b+94AryCcJs\nCiTYZCbosgIs2GQmyCcQHy+fLo9ltZqx2xt6+y2LdFJYWsee43aSos2MSVa3VkQEVKBJh1hrEP/r\njpF8uL2QL/YWs2lvMWNTI1gwMY5hcaHXLIwMBgMBPgEE+AQwhKguj33pm/+rircrOnKFdae6/QBq\n8jJ1Kt6iQiMwtfp1KuRCfYMxeWlNN3GNy7tclxdajU3tHa6mYxeoaqyloamR+uZGmq6zyxVsCrqi\ny2Xu6HR13vb39tNlxeJR3vn8OKDumYjI5VSgidOM0dHcNSeFT7cV8PnuIvbnVbI/r5KESDMLJsYx\ncaQNb69v/uHPaDA6P0zGMqTL/docbTQ0NV59aeUVhV3F+cr2F5Rf+zj+3n5dX1rZsR3sG4yPUX/9\npWeXulz1HZcWdtfpamy+3i5XELaOLld7kdX+e5ApiODLt3vocol4stPlDew4VMbQIcHcMlTdMxGR\nS/QJVTrx9jIyZVQUk0dGkl9Sx8bdRezLtbN6fQ7/+Cqf28bFMicjhiD/3v/QaDQYnYVUd1raWmho\nasQQ0MLJ8jOdunCXF3JnzlV0e5xAn4CuL628VMiZzLpj5QDTVZer8+WEHXO8rrPL5e/th9knCGtA\nBMEdhdalIuvyAiwpOopzda3qFIjQfudGUPdMRORKKtDkmgwGA6mxoaTGhmKvPc+mvcX882Apa/95\ngvU7TjItPYr5E+OIDg/s92zeRm/C/EKxhpsJbev6jl/Nrc3UNTVcc17cpe3qC7WUnj3T5TEMGAgy\nBXY5L659OwSzKVCXlrlQU2vT1zfN6Ci4nDfNuIkuV6R/RHuxdWXBdUXX63q7sUG+gZw3aF6XyOny\nBvbl2hkeH0Z6km5GIyJyORVo0iNrqD9Lb0vlnhlJbD1Uxhd7ivjqQClfHSjllqHhLJgYx6jEMLf7\nBtTHy4cIfwsR/t2f/C+0XKS+q7lxTZe6cXaKGku7PIbRYCTYZO7y0sp4Lxu1DedwOBw4cNDW8fvX\n223Onx0OB204cDjarti+9HzbNR5z0Ma1jnHZ9mXH62kfvxIfzp27eNU+V+b8+r20dfHersh6jfd2\nVRZH21W5rtzn8vdyoe0iF1su9vj3wd/bD7MpCFtAxNeXE15WZJl9Oi4vNAXhrzuLivSpD7cVAvCv\nt4/Q/2siIldQgSbXzd/XmwUT45g3Ppb9eXY27i7i8IkqDp+oIsYayPwJcUxNi8TH27MuCfTz9sXP\n24otwNrlPg6HgwutF77uwHVx58qSxlJONRT1Y3rPZsCAwWDA2PG7wfm7sfPjzp+Nl73GiMFowBIQ\ngp/R/6rO1uXb36TLJSJ969SZBvbnVZIcE0zGcCuVlY2ujiQi4lb0iUW+MaPRwPjhNsYPt1FYVs/n\nu4vYfayC//70GO9vKWBuRgxzx8USEjhw7qZoMBjw9/bH39ufqMDILvdzOBycbTl31by4Fq8mLpxv\nvqIIuaLwuGah0rFPp8Lkytcbr3G8r4sZ41XHM3Y6hvGyfS89ZgkLpK72/FV/tuHK19HxfJfvq73Y\nujrD17/fLN0OXsSzXOqe3TtjqLpnIiLXoAJNbkpSdDA/vDuNb81J5st9JWw5UMK67Sf5ZNcpJo+K\nZP6EOOIjza6O2W8MBgNBPoEE+QQSExTtfNzTighruBl7m+fkFRHPUFhWz4H8SlJiQxiVGObqOCIi\nbkkFmvQKS7Af35qTzF3TEtmRXcbGPcVsP3yG7YfPMDIhjPkT4xidHI5R35aKiAxa65zdM925UUSk\nKyrQpFf5mryYOy6W2RkxHC6oYuPuIo6equHoqRoiLQHMnxDL9PRofE2eNU9NRERuTmFZPQcLqhgW\nG8LIBHXPRES6ogJN+oTRYGBMSgRjUiIoqmjk891F7Mo5w/+/MZe1W04we+wQbhsfiyXYz9VRRUSk\nH1yae6Z1z0REuqcCTfpcnC2I7985kgfmJLN5XzGb95fwaeZpNmQVMWGElQUT4xk6pPvFqUVExHMV\nlNZxqKCKYXGhjFD3TESkWyrQpN+EBJq4d+ZQ7pyawK4j5WzcU0TW0QqyjlaQEhPCgolxZAyLwMuo\nBZ9FRAaSDzX3TETkuqlAk37n4+3FzDFDmDE6mpxTNXy+u4hDBVXkl9QRHuzHvAmxzBw9hAA//fUU\nEfF0+SV1ZJ+oZkS8umciItdDn4DFZQwGA2mJFtISLZRVneXzPcXsOFzGu1/m8+G2QmaMjmbehDhs\nof6ujioiIjfo8rlnIiLSMxVo4haiwwP57u3DuX/WULYcKGHT3mK+2FPMpr3FZKRaWTAxjtTYEF0a\nIyLiQfKL6zhSWM3IhDCGx6t7JiJyPVSgiVsJ8vfhzqmJ3D4pnt3HKti4u4h9uXb25dpJiDKzYGIc\nE0fY8PbSPDUREXf34bYTgLpnIiLfhAo0cUveXkampkUxZVQkecV1bNxdxP5cO6s/yuG9zfncNj6W\nB+YNd3VMERHpQl5xLUdO1jAqMYxhcaGujiMi4jFUoIlbMxgMDIsLZVhcKBW15/liTxFbD5Xx/pYT\nfLTjFNPSo5g/IZbo8EBXRxWRAey9995j3bp1zu3s7GzS09M5d+4cAQEBADz11FOkp6c792lububp\np5+mtLQULy8vXnzxReLi4vo9u6t8sFVzz0REboQKNPEYtlB//nXeMO6dMZSth0rZvL+Erzp+jU4O\nZ/7EOEYlhGmemoj0uiVLlrBkyRIAsrKy+PTTT8nPz+fFF19k2LBh13zN+vXrCQ4OZtWqVWzbto1V\nq1bxyiuv9Gdsl8ktquXoqRrSEsNIjVX3TETkm9BEHvE4AX7e3D4pnj/9Yh7L7k0nJSaEQwVVrHrn\nACv+nMXWg6U0t7S6OqaIDFCvv/46y5Yt63G/nTt3Mn/+fACmTZvGvn37+jqa2/hga8fcs5lDXZxE\nRMTzqIMmHsvLy8iEETYmjLBxorSejbtPs+eYnb98eoz3txQwJyOGueNiCQk0uTqqiAwQhw4dIjo6\nGqvVCsBvf/tbampqSE5OZvny5fj5+Tn3raysxGKxAGA0GjEYDDQ1NWEyDex/k46fruHY6VrSkyyk\nxIS4Oo6IiMdRgSYDwtAhwfz4nnSq515g095ithwoZd32k3yy6xRTRkWxYGIcsbYgV8cUEQ+3Zs0a\n7rvvPgC++93vMnz4cOLj41mxYgVvv/02jz76aJevdTgcPR4/LCwAb2+vm85ptZpv+hg36uX3DgLw\nyF1p15XDlVlvhCflVda+oax9x5Py9mVWFWgyoFiC/VgyN4W7piey/fAZvthTxLbDZWw7XMbIhDAW\nTIzjluRwjJqnJiI3IDMzk2eeeQbAefkiwK233sonn3zSaV+bzYbdbmfEiBE0NzfjcDh67J7V1Jy7\n6YxWqxm7veGmj3Mjjp2qIbugiluGhmMJ8Okxhyuz3ghPyqusfUNZ+44n5e2NrN0VeJqDJgOSn8mb\n28bH8vwPp/CTB0YzIj6Uo6dqeHXNIZ5ZncnmfcVcbNI8NRG5fuXl5QQGBmIymXA4HDzyyCPU19cD\n7YVbampqp/2nT5/OZ599BsDmzZuZPHlyv2fuTw6Hgw+26c6NIiI3Sx00GdCMBgNjUyMYmxrB6fIG\nPt9dRObRct7amMvaf55g9tgYbhsfS5jZ19VRRcTN2e1255wyg8HAgw8+yCOPPIK/vz+RkZH8+7//\nOwCPPfYYb7zxBosWLWLHjh089NBDmEwmfv3rX7syfp87dqqG3KJaRieHM3RIsKvjiIh4LBVoMmjE\nR5p5dPEovjUnmS/3lbB5fwmf7DrFhqzTTBxhY/7EOJKi9aFCRK4tPT2dN99807m9aNEiFi1adNV+\nb7zxBoBz7bPBQN0zEZHeowJNBp2QIF/umzWUO6cmsCunnM93F7Erp5xdOeWkxoawYGIcGalWjEbN\nUxMRuR45p2rIK65jTHK4vugSEblJKtBk0DL5eDFrzBBmjo7myMlqNu4uIvtENXnFdUSE+DFvQhwz\nR0fj76v/TUREuuJwOPhwa0f3bKa6ZyIiN0ufPGXQMxgMpCeFk54UTmnlWT7fU8SO7DO8symPD7ae\nYNaYIcwbH0tEqL+ro4qIuJ0jJ6vJL6ljbEoEiVHqnomI3CwVaCKXGRIRyPcWjuD+WUP56kApX+4r\nZuPuIj7fU8S4YVYWTIwjJSYEg27TLyLSuXumuWciIr1CBZrINZgDTNw1LZE7JseTdbScjbuL2Hvc\nzt7jdpKizcyfGMeE4Ta8vbRShYgMXkcKqykorScjNYKEKM9ZYFZExJ2pQBPphreXkWnp0UxNiyK3\nqJaNu4s4kFfJn9bl8J65gNvGxzJ77BAC/XxcHVVEpF/pzo0iIn1DBZrIdTAYDAyPD2N4fBjlNef4\nYk8x2w6VsearAtZtL2T6LdHMnxBHlCXA1VFFRPrF4RPVnCitZ9wwK/GR6p6JiPQWFWgi31BkWADf\nnj+M+2Ym8c+DZWzaW8TmfSVs3lfCmORwFkyMY0RCmOapiciA5XA4+HDbCUDdMxGR3qYCTeQGBfj5\nsHByPPMnxrL3uJ3PdxdxsKCKgwVVxFqDWDAxjsmjIvHx1jw1ERlYDhVUUVjWwPjhVuJsQa6OIyIy\noKhAE7lJXkYjk0ZGMmlkJAUldXy+p4g9x+z8+ZOjrNlSwK0ZMTwwb7irY4qI9Ir27lnH3LPp6p6J\niPQ2FWgivSg5JoTkmBCq5lxg075ithwo5YNthazfeYqRCWFkpEYwNjWC0CBfV0cVEbkhB/OrOHmm\ngQkjbMSqeyYi0utUoIn0gfAQPx6cm8Ld0xPZfvgM27PPcPhEFYdPVPG3DccZOiSYjNQIMlKtRIcH\naL6aiHiES90zA3DP9ERXxxERGZBUoIn0IT+TN7eNj2XpwpHk5FVwIK+S/Xl2covqOFFaz/tbThBp\nCSAjNYJxqVaGDgnGaFSxJiLu6UB+JafKG5g00kaMVd0zEZG+cF0F2gsvvMDBgwcxGAwsX76c0aNH\nX7XPqlWrOHDgAG+99VavhxQZCKyh/syfGMf8iXE0nm/mUEEl+3MrOVxYxWeZp/ks8zTBAT6MSYkg\nY5iVUQlhmHy8XB1bRATo3D27S3PPRET6TI8FWlZWFqdOneLdd9+loKCA5cuX8+6773baJz8/n927\nd+Pjo8V6Ra5HkL8P09KjmZYeTVNzKzmnajiQZ+dAXiVbD5Wx9VAZJh8jtySFMzY1gjEpEQT56/8v\nEXGd/XmVnC5vbO+eRQS6Oo6IyIDVY4G2c+dO5s2bB0BycjJ1dXU0NjYSFPT1pQ2//vWv+elPf8rv\nfve7vksqMkCZfLwYmxLB2JQI2m53cKK0nv15dvblVbI3187eXDtGg4FhcSGMTbWSkRqBNdTf1bFF\nZBBpu6x7dre6ZyIifarHAq2yspK0tDTntsViwW63Owu0tWvXMmnSJGJiYvoupcggYTQaSIkNISU2\nhCVzUyirOsu+3PbO2rHTtRw7Xcs7m/KItQa1z1sbZiU+Mkg3GRGRPrU/105RRSNTRkUyRN0zEZE+\n9Y1vEuJwOJw/19bWsnbtWv7yl79QXl5+Xa8PCwvA2/vm59VYreabPkZ/8qS8yto3biSr1Wpm9Igo\nAKrrL5B15AyZR85wINfORztO8tGOk0SE+jMlLYrJ6VGkJ0fg7dU7C2MP9LF1FWUVT+PsnhngLt25\nUUSkz/VYoNlsNiorK53bFRUVWK1WAHbt2kV1dTXf/va3aWpq4vTp07zwwgssX768y+PV1Jy76dBW\nqxm7veGmj9NfPCmvsvaN3so6PiWc8SnhnL/YwpHCavbn2TmYX8X67YWs315IgK83o5PDyRhmJT3J\ngr/vjd2odTCObX8YjFlV5Hm+fcftFNvPMjUtkuhwdc9ERPpaj5/epk+fzmuvvcbSpUs5cuQINpvN\neXnjwoULWbhwIQDFxcX84he/6LY4E5He4e/rzYQRNiaMsNHS2kZeUS378io5kGdnV045u3LK8fYy\nMCIhjHGpVi2OLSI3pM3h4MPtl7pnmnsmItIfeizQxo0bR1paGkuXLsVgMLBixQrWrl2L2Wxm/vz5\n/ZFRRLrh7WVkZKKFkYkW/nVeKkUVjc55a9knqsk+Uc3fNhwnKTqYccMiGJtqZYgWxxaR67D3uJ0S\n+1mmpUcRZQlwdRwRkUHhuq5/+tnPftZpe8SIEVftExsbqzXQRFzMYDAQH2kmPtLMvTOHUll7nv35\nlezPbV8cu7CsY3HsMH8yUq1kDIsgeUiIFscWkau0ORys21aI0WDQ3DMRkX50YxNURMQjRIT6M39C\nHPMnXLY4dkdn7bOs03yWdRpzx+LY41KtjErU4tgi0m7PsQpKKs8yPT2KyDB1z0RE+osKNJFB4vLF\nsZtbWsk5WcP+vEoO5Fey7VAZ2zoWx05PCmfWuFiGRgZpcWyRQaqtrf3OjeqeiYj0PxVoIoOQj7cX\nY1IiGJMSQZvj68Wx9+dWsi/Xzr5cOwYDDIsNJSM1grHDrNi0OLbIoJF1rJyyqnPMuCUam7pnIiL9\nSgWayCBnNBhIiQkhJSaEJXPaF8fOLW1g24FicotqOV5Uyztf5hNrDXTOW0uINOsmIyIDVFubg4+2\nn8TLaGCxumciIv1OBZqIdBIdHsjoEVHMviWKusaLHMhvn7eWc7LGuTh2mNmXjNQIMlKtDI8P7bXF\nsUXc1Xvvvce6deuc29nZ2fz973/n2WefxWg0EhwczKpVq/D3/7rTvHbtWl599VXi4+MBmDZtGo89\n9li/Z/+mso62d89mjo5W51xExAVUoIlIl0KCfJk9NobZY2O40NRC9on2xbEPFVTx5b4SvtxXgv+l\nxbFTI7hlaPgNL44t4s6WLFnCkiVLAMjKyuLTTz/lueee4+mnn2b06NG89NJLrF27lm9/+9udXrdo\n0SKeeuopV0S+IW1tDtZ1dM/umpbo6jgiIoOSPkmJyHXxM129OPb+vPbuWmZOOZk55XgZDYxMCCNj\nmJWxKRGEmbU4tgw8r7/+OitXrsTf35+goCAALBYLtbW1Lk528zJzyjlTfY5ZY4YQoe6ZiIhLqEAT\nkW/s8sWxH+pYHLu9WLOTXVhNdmE1b204TlK0uX3eWmoEQyICNW9NPN6hQ4eIjo7GarU6Hzt37hwf\nfvghr7766lX7Z2Vl8eijj9LS0sJTTz3FqFGjuj1+WFgA3t43v9SF1Wr+xq9pbW3j412n8DIa+O7i\nNKz9tDD1jWR1JU/Kq6x9Q1n7jifl7cusKtBE5KZcvjj2PTOSqKw7z4GOztrx07UUljWw9p8nsIX5\nO+etpcRocWzxTGvWrOG+++5zbp87d47HHnuM73//+yQnJ3fad8yYMVgsFubMmcP+/ft56qmn+Oij\nj7o9fk3NuZvOaLWasdsbvvHrth8uo7TyLLPHDsHY2npDx/imbjSrq3hSXmXtG8radzwpb29k7a7A\nU4EmIr0qIsSfeRPimNexOPbhgir259k5fKKaDVlFbMgqci6OnZEaQVqiRYtji8fIzMzkmWeeAaCl\npYVly5axePFi7r///qv2TU5OdhZtGRkZVFdX09raipeX+/19b21r46MdHXdunJro6jgiIoOaCjQR\n6TNB/j5MTY9ianoUzS2tHD1V45y35lwc29tIWpKFjFQrY1LCMQeYXB1b5JrKy8sJDAzEZGr/O7p6\n9WomTZrkvHnIlVavXk10dDSLFy8mNzcXi8XilsUZwM7scipqzjMnI4bwED9XxxERGdRUoIlIv/Dx\n9mJ0cgSjkyP4zu0OCkvrnfPWLhVtBgOkdiyOnZEaoQVyxa3Y7XYsFotz++233yY2NpadO3cCMHny\nZJ544gkee+wx3njjDe666y5+/vOf884779DS0sLzzz/vqujdamlt46MdhXh7GVg8NcHVcUREBj0V\naCLS74wGA8kxISTHhPCtOcmUVZ11zlvLK6olt6iWd7/MJ+bS4tipESRGaXFsca309HTefPNN5/a2\nbduuud8bb7wBQFRUFG+99Va/ZLsZO4+cwV57gbnjYrAEq3smIuJqKtBExOWiwwOJDg/kjikJ1J1t\n4mB+Jftz7Rw5WcP6HSdZ37E49tiOztqI+DAtji3SC1pa2/ho+0m8vQzcOUXdMxERd6ACTUTcSkig\niVljhjBrzBAuI3VBRAAAEkRJREFUNLVwpLCafbmVHCqoZPO+EjbvK8Hf14tbhoaTkWplziR94y9y\no3Zkn6Gy7gK3jYtV90xExE2oQBMRt+Vn8mb8cBvjh9tobWsjt6iO/Xl2DuRVknW0gqyjFaxen8PQ\n6GDSkiykJ1lIjDbjZVR3TaQnLa1trN9xEm8vI4s090xExG2oQBMRj+BlNDIyIYyRCWE8dFv74tgH\n8io5VlTL8dM15JfU8eG2QgJ8vRmZGEZ6koW0JAsRIf6uji7ilrYfLmvvno2PJczs6+o4IiLSQQWa\niHicyxfHftRq5lRRNUdP1ZBdWE32iWr2Hrez97gdgEhLAOmJFtKGWhgRH4qfSf/siVzqnvl4G1mk\nuWciIm5Fn1RExOMF+Pk4L4V0OBxU1Jwnu7CaI4XVHD1dw6Z9xWzaV4yX0UBKTAjpQ9u7a/GRZoy6\nM6QMQtsOlVFVf5F5E9Q9ExG55KuvNjFnzm097vf888+zePEDDBkS0yc5VKCJyIBiMBiItAQQaQng\ntvGxtLS2UVBS5yzYcotqOV5Uy/tbThDk70NakoW0xPaCTR9UZTBoaW1j/U51z0RELldWVsoXX2y4\nrgLtl7/8JXZ7Q59lUYEmIgOat5eR4fFhDI8P44HZyTScayLnZA1HCqs5crKazJxyMnPKAYixBpKW\n2H6zkWFxoZh8vFycXqT3bT1URnX9RRZMjCM0SF9KiIgAvPzySxw9eoSZMyeyYMEdlJWV8sorv+fF\nF5/Fbq/g/PnzfP/7P2T69Jl85zvf4Ykn/jebN2/i7NlGTp8+RUlJMT/5yZNMnTr9prOoQBORQcUc\nYGLyqEgmj4rE4XBQWnmWI4XVZJ+sJvd0LRvtRWzcXdRe2MWFkJYUTlqShVhroBbKFo/X3NI+98zk\nbeSOyfGujiMick3/+DKf3ccqevWYE0fYePDWlC6ff+ih77B27T9ISkrm9OmT/P73b1JTU82kSVO4\n447FlJQU86tfPc306TM7va6iopyVK3/Lrl07+PDD91WgiYjcDIPBQIw1iBhrEAsmxdPc0kpucV17\nd62wmiMnazhysgY2t6/PltZxZ8i0RAvBgSZXxxf5xrYeKqWmob17FqLumYjINY0cmQaA2RzM0aNH\nWLduLQaDkfr6uqv2HT16LAA2m43GxsZe+fNVoImIdPDx9mqfj5ZogblQ23iRnJPVzoJtR/YZdmSf\nASA+Mqhj7bVwUmJC8PHW2mvi3ppbWvl456n27pnmnomIG3vw1pRuu119zcfHB4DPP/+M+vp6Xn/9\nTerr6/nBD75z1b5eXl9Ph3A4HL3y56tAExHpQmiQL9PSo5mWHk2bw0FxRWP75ZCF1eQV13K6vJFP\nd53G5GNkRHxY+/y1oRaiLAG6HFLczj8PllHTcJGFk+IJUQdYRKQTo9FIa2trp8dqa2uJjh6C0Whk\ny5YvaW5u7pcsKtBERK6D8bK11+6YksDFplaOF9V2FGxVHCpo/8UmsAT7dhRr4YxMCCPI38fV8WWQ\na++encTkY2Sh5p6JiFwlISGJ48ePER09hNDQUADmzLmVp5/+3+TkZHPnnXdjs9n4y19W93kWFWgi\nIjfA1+TF6ORwRieHA6lU119wdtdyTlaz9VAZWw+VYQASo4NJS7IwIyMWS4A33l66HFL611cHSqlt\nbOKOyfGaPykicg1hYWGsXftxp8eio4fw17++49xesOAOAKxWM3Z7A0OHfn0Z5tChKfzud3/qlSwq\n0EREeoEl2I+ZY4Ywc8wQ2tocnCpvIPtEFUcKqykoraewrJ71O07iZ/JiZEIY6R03HLGFBbg6ugxw\nTc2tfLLrFL4+XuqeiYh4ABVoIiK9zGg0kBQdTFJ0MHdNT+L8xRaOna6hoKyBPUfL2Z9Xyf68SgCs\noX6kd9zKf0R8GAF++mdZeteWA6XUNTaxaEoC5gB1z0RE3J0+CYiI9DF/X28yUq0smDYUu72Bitrz\nzjtDHj1Vzeb9JWzeX4LRYCA5Jth5O/+kqGCMRt1sRG6cs3tm8uL2SXGujiMiItdBBZqISD+zhfpj\ny4hhbkYMrW1tnCitdxZs+SV15BXX8cHWQgL9vBmZaGm/HDLRQniIn6uji4f5an8JdWebuHOqumci\nIp5CBZqIiAt5GY2kxoaSGhvKvTOHcvZCM0dP1pBdWM2Rwir2HKtgz7EKAKLDA9rXaeu4HNLX5NXD\n0WUwu9jcyieZp/EzeXH7JM09ExHxFCrQRETcSKCfDxNG2JgwwobD4eBM9Tlnd+3Y6Vq+2FvMF3uL\n8TIaSI0NcS6WHRcZhFFrr8llNu8rof5sE4unJWipBxERD6ICTUTETRkMBqLDA4kOD2TehDiaW9oo\nKKnjyMlqsk+0F2zHTtfy/pYTmAN8nN21tCQLoUG+ro4vLnSxqZVPM0/h7+vFgonqnomI9JZvfesu\nPvnk4553vAkq0EREPISPt5ERCWGMSAjjgdnJ1J9rIudkNUdOVJN9sppdOeXsyikHINYa6OyupcaG\nYPLR5ZCDyeb9JTSca+auaYnqnomIeBgVaCIiHio4wMSUUVFMGRWFw+GgpPKsc7Hs3KJaiu1FbMgq\nwsfbyLC4UOfaazERgRh0OeSAdeFiS0f3zJsFunOjiMh1+f73v80LL6wiKiqKM2fK+MUvnsRqtXH+\n/HkuXLjAT3/6c0aNSu+XLCrQREQGAIPBQKw1iFhrELdPiqepuZW84jqyC6ucc9iOFFYDEBpkcl4K\nOSrRQrDu7jegfLy9kIZzzdw9PZFAP3XPRMTzrM1fz/6Kw716zAzbLdyfsrjL52fNmsv27f/kgQce\nZOvWLcyaNZfk5FRmzZrD3r27efvtv/L88/+nVzN1RQWaiMgAZPLxchZhADUNF9svhyys5sjJarYf\nPsP2w2cASIg0kz60/Vb+KbEheHsZXRldbsKFphbWfpXf3j2bqO6ZiMj1mjVrLr/73Ss88MCDbNu2\nhSee+CnvvPMWf//7WzQ3N+Pn139L3ahAExEZBMLMvky/JZrpt0TT5nBQVN7o7K7lFddxqryBj3ee\nwtfHixHxocwaH8eYpDDdGbLDe++9x7p165zb2dnZ/P3vf+c///M/ARg+fDj/9V//1ek1zc3NPP30\n05SWluLl5cWLL75IXFzfFk2b9hZTf7aJe2YkEaDumYh4qPtTFnfb7eoLQ4cmU1Vlp7z8DA0NDWzd\n+hURETZ+9av/j2PHcvjd717ptywq0EREBhmjwUBClJmEKDN3Tk3kQlMLx0/XOrtrBwuqOFhQxa9/\nNAVbWICr47qFJUuWsGTJEgCysrL49NNPef7551m+fDmjR4/mySefZMuWLcyePdv5mvXr1xMcHMyq\nVavYtm0bq1at4pVX+vYEf6SwGnOAD/MnqHsmIvJNTZ06gz/96ffMnDmb2toakpNTAdiyZTMtLS39\nlkMFmojIIOdn8mZMSgRjUiIAqKq7AD5ehAeoA3Mtr7/+Oi+++CIPP/wwo0ePBmDu3Lns3LmzU4G2\nc+dO7r33XgCmTZvG8uXL+zzbDxaPIiQ0AK+2tj7/s0REBprZs+fy4x9/n//+779z4cJ5nntuBZs3\nf8EDDzzIF19s5OOP1/V8kF6gAk1ERDoJD/HDajVjtze4OorbOXToENHR0Xh5eREcHOx8PDw8HLvd\n3mnfyspKLJb2OYBGoxGDwUBTUxMmU9c3ZQkLC8Db+8aXRLBazTf8WlfxtMyelFdZ+4ay9h1X57Va\np5CTk+Pc3rhxg/Pn++5rv+TykUe+DUBgYN/lUIEmIiJyndasWcN999131eMOh6PH117PPjU1524o\n1+U8qbj2pKzgWXmVtW8oa9/xpLy9kbW7YlS36hIREblOmZmZZGRkYLFYqK2tdT5eXl6OzWbrtK/N\nZnN21Zqbm3E4HN12z0REREAFmoiIyHUpLy8nMDAQk8mEj48PQ4cOZc+ePQBs3LiRmTNndtp/+vTp\nfPbZZwBs3ryZyZMn93tmERHxPCrQREREroPdbnfOKQNYvnw5L7/8MkuXLiU+Pp5p06YB8NhjjwGw\naNEi2traeOihh3j77bd58sknXZJbREQ8i+agiYiIXIf09HTefPNN53ZKSgr/8z//c9V+b7zxBoBz\n7TMREZFvQh00ERERERERN6ECTURERERExE2oQBMREREREXETKtBERERERETchAo0ERERERERN2Fw\nOBwOV4cQERERERERddBERERERETchgo0ERERERERN6ECTURERERExE2oQBMREREREXETKtBERERE\nRETchAo0ERERERERN+Ht6gA9eeGFFzh48CAGg4Hly5czevRo53M7duzg5ZdfxsvLi1mzZvH444+7\nMGn3WW+99VaioqLw8vICYOXKlURGRroqKgC5ubksW7aMRx55hIcffrjTc+42tt1ldbex/c1vfsPe\nvXtpaWnhRz/6EQsWLHA+527j2l1WdxrX8+fP8/TTT1NVVcXFixdZtmwZc+fOdT7vTuPaU1Z3GtfL\nXbhwgcWLF7Ns2TLuv/9+5+PuNLbSmSedH8GzzpE6P/YdnSN7n86Rfcsl50eHG8vMzHT88Ic/dDgc\nDkd+fr7jwQcf7PT8HXfc4SgtLXW0trY6HnroIUdeXp4rYjocjp6zzp0719HY2OiKaNd09uxZx8MP\nP+x45plnHG+99dZVz7vT2PaU1Z3GdufOnY4f/OAHDofD4aiurnbMnj270/PuNK49ZXWncf34448d\nf/rTnxwOh8NRXFzsWLBgQafn3Wlce8rqTuN6uZdfftlx//33O95///1Oj7vT2MrXPOn86HB41jlS\n58e+o3Nk39A5sm+54vzo1pc47ty5k3nz5gGQnJxMXV0djY2NABQVFRESEkJ0dDRGo5HZs2ezc+dO\nt8zqjkwmE6tXr8Zms131nLuNbXdZ3c3EiRN59dVXAQgODub8+fO0trYC7jeu3WV1N4sWLeLf/u3f\nACgrK+v0bZq7jWt3Wd1VQUEB+fn5zJkzp9Pj7ja28jVPOj+CZ50jdX7sOzpH9g2dI/uOq86Pbn2J\nY2VlJWlpac5ti8WC3W4nKCgIu92OxWLp9FxRUZErYgLdZ71kxYoVlJSUMH78eJ588kkMBoMrogLg\n7e2Nt/e1//O729h2l/USdxlbLy8vAgICAFizZg2zZs1ytundbVy7y3qJu4zrJUuXLuXMmTP84Q9/\ncD7mbuN6ybWyXuJu4/rSSy/xq1/9ig8++KDT4+46tuJZ50fwrHOkzo99R+fIvqVzZO9z1fnRrQu0\nKzkcDldHuG5XZv3JT37CzJkzCQkJ4fHHH2fDhg0sXLjQRekGFncc2y+++II1a9bw5z//2aU5rkdX\nWd1xXN955x2OHj3Kz3/+c9atW+fyk2F3usrqbuP6wQcfMHbsWOLi4lyWQW6eJ50fQefI/uKu46pz\nZN/QObJ3ufL86NaXONpsNiorK53bFRUVWK3Waz5XXl7u0hZ/d1kB7r33XsLDw/H29mbWrFnk5ua6\nIuZ1cbex7Ym7je3WrVv5wx/+wOrVqzGbzc7H3XFcu8oK7jWu2dnZlJWVATBy5EhaW1uprq4G3G9c\nu8sK7jWuAF999RWbNm3iwQcf5L333uP3v/89O3bsANxvbOVrnnR+hIFzjnTHse2OO46rzpG9T+fI\nvuHK86NbF2jTp09nw4YNABw5cgSbzea8HCI2NpbGxkaKi4tpaWlh8+bNTJ8+3S2zNjQ08Oijj9LU\n1ATA7t27SU1NdVnWnrjb2HbH3ca2oaGB3/zmN/zxj38kNDS003PuNq7dZXW3cd2zZ4/z28vKykrO\nnTtHWFgY4H7j2l1WdxtXgFdeeYX333+ff/zjHyxZsoRly5Yxbdo0wP3GVr7mSedHGDjnSHcc2664\n47jqHNk3dI7sG648Pxocbn5dxMqVK9mzZw8Gg4EVK1aQk5OD2Wxm/vz57N69m5UrVwKwYMECHn30\nUbfN+te//pUPPvgAX19fRo0axa9+9SuXtp6zs7N56aWXKCkpwdvbm8jISG699VZiY2Pdbmx7yupO\nY/vuu+/y2muvkZSU5Hxs8uTJDB8+3O3Gtaes7jSuFy5c4Je//CVlZWVcuHCBJ554gtraWrf8t6Cn\nrO40rld67bXXiImJAXDLsZXOPOn8CJ5zjtT5se/oHNk3dI7se/19fnT7Ak1ERERERGSwcOtLHEVE\nRERERAYTFWgiIiIiIiJuQgWaiIiIiIiIm1CBJiIiIiIi4iZUoImIiIiIiLgJFWgiIiIiIiJuQgWa\niIiIiIiIm1CBJiIiIiIi4ib+H73+Z5wGDsqVAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7fb0175d8a58>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "4EmFhiX-FMaV",
        "colab_type": "code",
        "outputId": "42d7dcac-f4cb-4bf2-e0e9-c3ee2007df1d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# Test performance\n",
        "trainer.run_test_loop()\n",
        "print(\"Test loss: {0:.2f}\".format(trainer.train_state['test_loss']))\n",
        "print(\"Test Accuracy: {0:.1f}%\".format(trainer.train_state['test_acc']))"
      ],
      "execution_count": 92,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Test loss: 0.48\n",
            "Test Accuracy: 83.2%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "zVU1zakYFMVF",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Save all results\n",
        "trainer.save_train_state()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qLoKfjSpFw7t",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Inference"
      ]
    },
    {
      "metadata": {
        "id": "tDu1oFRsWPlr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class Inference(object):\n",
        "    def __init__(self, model, vectorizer, device=\"cpu\"):\n",
        "        self.model = model.to(device)\n",
        "        self.vectorizer = vectorizer\n",
        "        self.device = device\n",
        "  \n",
        "    def predict_category(self, dataset):\n",
        "        # Batch generator\n",
        "        batch_generator = dataset.generate_batches(\n",
        "            batch_size=len(dataset), shuffle=False, device=self.device)\n",
        "        self.model.eval()\n",
        "        \n",
        "        # Predict\n",
        "        for batch_index, batch_dict in enumerate(batch_generator):\n",
        "            # compute the output\n",
        "            y_pred =  self.model(batch_dict['title'], batch_dict[\"title_length\"],\n",
        "                                 apply_softmax=True)\n",
        "\n",
        "            # Top k nationalities\n",
        "            y_prob, indices = torch.topk(y_pred, k=len(self.vectorizer.category_vocab))\n",
        "            probabilities = y_prob.detach().to('cpu').numpy()[0]\n",
        "            indices = indices.detach().to('cpu').numpy()[0]\n",
        "\n",
        "            results = []\n",
        "            for probability, index in zip(probabilities, indices):\n",
        "                category = self.vectorizer.category_vocab.lookup_index(index)\n",
        "                results.append({'category': category, 'probability': probability})\n",
        "\n",
        "        return results"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "KGsxAknUWjgT",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Load vectorizer\n",
        "with open(args.vectorizer_file) as fp:\n",
        "    vectorizer = NewsVectorizer.from_serializable(json.load(fp))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "dvkAgy6mWjdC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "outputId": "1c63df18-817b-4435-c3c4-4a4116529822"
      },
      "cell_type": "code",
      "source": [
        "# Load the model\n",
        "model = NewsModel(embedding_dim=args.embedding_dim, \n",
        "                  num_embeddings=len(vectorizer.title_vocab), \n",
        "                  rnn_hidden_dim=args.rnn_hidden_dim,\n",
        "                  hidden_dim=args.hidden_dim,\n",
        "                  output_dim=len(vectorizer.category_vocab),\n",
        "                  num_layers=args.num_layers,\n",
        "                  bidirectional=args.bidirectional,\n",
        "                  dropout_p=args.dropout_p, \n",
        "                  pretrained_embeddings=None, \n",
        "                  padding_idx=vectorizer.title_vocab.mask_index)\n",
        "model.load_state_dict(torch.load(args.model_state_file))\n",
        "print (model.named_modules)"
      ],
      "execution_count": 104,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<bound method Module.named_modules of NewsModel(\n",
            "  (embeddings): Embedding(3406, 100, padding_idx=0)\n",
            "  (gru): GRU(100, 128, batch_first=True)\n",
            "  (dropout): Dropout(p=0.1)\n",
            "  (fc1): Linear(in_features=128, out_features=100, bias=True)\n",
            "  (fc2): Linear(in_features=100, out_features=4, bias=True)\n",
            ")>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "oUBYxSEfWjZy",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Initialize\n",
        "inference = Inference(model=model, vectorizer=vectorizer, device=args.device)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "KXHyLE_PWjXZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class InferenceDataset(Dataset):\n",
        "    def __init__(self, df, vectorizer):\n",
        "        self.df = df\n",
        "        self.vectorizer = vectorizer\n",
        "        self.target_size = len(self.df)\n",
        "\n",
        "    def __str__(self):\n",
        "        return \"<Dataset(size={1})>\".format(self.target_size)\n",
        "\n",
        "    def __len__(self):\n",
        "        return self.target_size\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        row = self.df.iloc[index]\n",
        "        title_vector, title_length = self.vectorizer.vectorize(row.title)\n",
        "        return {'title': title_vector, 'title_length': title_length}\n",
        "\n",
        "    def get_num_batches(self, batch_size):\n",
        "        return len(self) // batch_size\n",
        "\n",
        "    def generate_batches(self, batch_size, shuffle=True, drop_last=False, device=\"cpu\"):\n",
        "        dataloader = DataLoader(dataset=self, batch_size=batch_size, \n",
        "                                shuffle=shuffle, drop_last=drop_last)\n",
        "        for data_dict in dataloader:\n",
        "            out_data_dict = {}\n",
        "            for name, tensor in data_dict.items():\n",
        "                out_data_dict[name] = data_dict[name].to(device)\n",
        "            yield out_data_dict"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3FXcmtXWWjUY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "4a7e757d-b1e9-42c8-8f84-ac7029c2146f"
      },
      "cell_type": "code",
      "source": [
        "# Inference\n",
        "title = input(\"Enter a title to classify: \")\n",
        "infer_df = pd.DataFrame([title], columns=['title'])\n",
        "infer_df.title = infer_df.title.apply(preprocess_text)\n",
        "infer_dataset = InferenceDataset(infer_df, vectorizer)\n",
        "results = inference.predict_category(dataset=infer_dataset)\n",
        "results"
      ],
      "execution_count": 110,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Enter a title to classify: President Obama signed the petition during the White House dinner.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'category': 'World', 'probability': 0.40057328},\n",
              " {'category': 'Business', 'probability': 0.30899048},\n",
              " {'category': 'Sci/Tech', 'probability': 0.27638948},\n",
              " {'category': 'Sports', 'probability': 0.01404677}]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 110
        }
      ]
    },
    {
      "metadata": {
        "id": "noPtpaHZ6NAW",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Layer normalization"
      ]
    },
    {
      "metadata": {
        "id": "t3bu5cEP6PSb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Recall from our [CNN notebook](https://colab.research.google.com/github/GokuMohandas/practicalAI/blob/master/notebooks/11_Convolutional_Neural_Networks.ipynb) that we used batch normalization to deal with internal covariant shift. Our activations will experience the same issues with RNNs but we will use a technique known as [layer normalization](https://arxiv.org/abs/1607.06450) (layernorm) to maintain zero mean unit variance on the activations. \n",
        "\n",
        "With layernorm it's a bit different from batchnorm. We compute the mean and var for every single sample (instead of each hidden dim) for each layer independently and then do the operations on the activations before they go through the nonlinearities. PyTorch's [LayerNorm](https://pytorch.org/docs/stable/nn.html#torch.nn.LayerNorm) class abstracts all of this for us when we feed in inputs to the layer."
      ]
    },
    {
      "metadata": {
        "id": "-G-mVmUL61Fk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "$ LN = \\frac{a - \\mu_{L}}{\\sqrt{\\sigma^2_{L} + \\epsilon}}  * \\gamma + \\beta $\n",
        "\n",
        "where:\n",
        "* $a$ = activation | $\\in \\mathbb{R}^{NXH}$ ($N$ is the number of samples, $H$ is the hidden dim)\n",
        "* $ \\mu_{L}$ = mean of input| $\\in \\mathbb{R}^{NX1}$\n",
        "* $\\sigma^2_{L}$ = variance of input | $\\in \\mathbb{R}^{NX1}$\n",
        "* $epsilon$ = noise\n",
        "* $\\gamma$ = scale parameter (learned parameter)\n",
        "* $\\beta$ = shift parameter (learned parameter)"
      ]
    },
    {
      "metadata": {
        "id": "P0e9TnQ581-1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/layernorm.png\" width=400>"
      ]
    },
    {
      "metadata": {
        "id": "oAhAHcgZBMFe",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The most useful location to apply layernorm will be inside the RNN on the activations before the non-linearities. However, this is a bit involved and though PyTorch has a [LayerNorm](https://pytorch.org/docs/stable/nn.html#torch.nn.LayerNorm) class, they do not have an RNN that has built in layernorm yet. You could implement the RNN yourself and manually add layernorm by following a similar setup like below.\n",
        "\n",
        "```python\n",
        "# Layernorm\n",
        "for t in range(seq_size):\n",
        "    # Normalize over hidden dim\n",
        "    layernorm = nn.LayerNorm(args.hidden_dim)\n",
        "    # Activating the module\n",
        "    a = layernorm(x)\n",
        "```"
      ]
    },
    {
      "metadata": {
        "id": "1YHneO3SStOp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# TODO"
      ]
    },
    {
      "metadata": {
        "id": "gGHaKTe1SuEk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "- interpretability with task to see which words were most influential"
      ]
    }
  ]
}